7043SCN Generative AI and Reinforcement Learning Assignment 1 — Comparative Fine-tuning of Phi-3 Mini for E-commerce Product Question Answering

Student: J. Panfilova
Module:7043SCN Generative AI & Reinforcement Learning
Academic year 2026/27
Task 1 – LLM Fine-Tuning
Status: Milestone 1 — Project set-up and baseline preparation

Project Summary

This project examines the likelihood of improving a publicly available Large Language Model (LLM) on factual question answering about e-commerce product catalogs through parameter-efficient finetuning.

The selected tool is "eBay product question answering" where the model comes with a product listing and needs to answer questions like the following:

Does the product exist at the moment?
You discontinued the product?
Is the product listed as used?
What's the best the asking price?

The research question is:

How much does parameter-effective fine-tuning boost the question response accuracy of the Phi-3 Mini on eBay product listings over the pretrained model?

The proposed high-level fine-tuning technique is QLoRA, which is built into the existing GPU memory.

MODELING

The selected Hugging Face model is:

microsoft/Phi-3-mini-4k-instruct

Before fine-tuning, we use a decoder-only instruction-following language model as the pre-trained baseline model.

The model is loaded with the data that follows over current tests:

Quantization of 4 bits
Quantization of NF4
Double quantization
FP16 calculation bitsandbytes
PEFT/LoRA for the desired QLoRA experiment

The model was successfully loaded and tested on the available GPU (NVIDIA Tesla T4).

3.1 Dataset

The project uses the public dataset from Hugging Face: montassar-souli/ebay_productlisting

The original dataset of 30,000 product listings and 26 columns.

Over the initial processing of data, six entirely empty columns were excluded. Additionally, 448 broken links entries were removed, resulting in 29,552 valid product listings.

Seed number 42 was utilized to produce an 80/10/10 split at the item level:

Split > Products > Train 23639 > Validation 2955 > Test 2956

The three splits do not have overlapping product IDs.

Eight factual QA examples are generated for each product using:

Create Model Name
Stock.
Pre-owned
Out of stock
Cost
Number of reviews from seller
Seller Ranking

Missing values are represented as "Not specified"

Appraisal

An evaluation set of 800 held out test examples has been frozen:

200 Out of Production
Cost 200
200 Shares
Used 200
200 different products

We will use the same frozen evaluation set for both the pretrained baseline and the QLoRA model.

The analysis relies on semantic metrics involving Exact Match, Token F1 and ROUGE-L. Natural-language responses could be correct paraphrases of the raw dataset parameters, therefore a task-specific **actual accuracy ** metric also serves as the main metric.

Environment and hardware

Experiments are conducted on Google Colab using:

GPU: NVIDIA Tesla T4
Graphics Processing Unit (GPU) memory: 14.56 GB
CUDA: 13.0
PyTorch version: 2.11.0+cu130
Transformers: 5.17.0
PEFT: 0.21.0
bitsandbytes: 0.50.2

Limited GPU memory makes parameter-efficient fine-tuning particularly relevant. T4 was successfully tested with 4-bit NF4 load.

Status at Baseline

The pre-trained Phi-3 model was assessed on the complete set of 800 frozen test instances.

Task-specific factual accuracy (corrected):

Category > Accuracy ------: Removed > 33.50% > Price > 63.00% > Stock > 94.50% > Used > 36.50% > Overall > 56.88%

The predictions and evaluation results for the baseline have been saved in the project results directory.

An evaluation bug on Stock category was found and fixed before freezing the final baseline. The predictions of the model were not redone; only the evaluation procedure was fixed.

What’s Next?

Next stage of the project will be:

Make Training and Validation Prompts.
Install QLoRA with PEFT and bitsandbytes.
Check trainable LoRA parameters and GPU memory usage.
Perform a brief training smoke test.
Train QLoRA model.
Test QLoRA on the same frozen test set of 800 examples.
Quantitative and qualitative comparison of baseline and QLoRA.
Errors, constraints, reliability, and ethical/professional concerns.

All relevant datasets, test documents, and experimental metadata and outcomes are kept in the project directory for reproducibility purposes.

panfilovaj/7043SCNGenerativeAI2627SEPNOV

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Languages