NTU SC3000 Assignment 1
nanoGPT Arithmetic Reasoning with DPO
Fine-tuning a pretrained nanoGPT checkpoint with Direct Preference Optimization to answer single-step arithmetic and algebra prompts in a consistent response format.
This NTU SC3000 artificial intelligence project adapts a pretrained nanoGPT checkpoint for single-step arithmetic and simple algebra. The model learns to repeat the prompt, state the answer, and justify it in one consistent sentence.
Goal
The target response follows the repository’s exact structure: PROMPT The answer is A because B is C. Preference pairs teach the model to choose the correct, formatted response over an unhelpful negative response.
Method
Direct Preference Optimization (DPO) is the only additional fine-tuning method used. Starting from the provided pretrained nanoGPT checkpoint, we trained with generated positive and negative arithmetic responses using AdamW and a cosine-annealing learning-rate schedule.
Model output
A successful output from the final notebook keeps the question and answer together on one line:
17+19=? The answer is 36 because 17+19 equals 36.
The model reached 75% accuracy on the provided test set and 87% on the held-out custom test set. The results also exposed a clear limitation: it learned the requested format reliably, but could still make arithmetic errors on some inputs.
