Pre-training & Fine-tuning

Dataset Scale & Sources

Modern LLMs are trained on trillions of tokens drawn from diverse sources:

• Common Crawl: petabyte-scale web text snapshots — largest source but noisiest
• Books (Books3, Gutenberg): long-form coherent reasoning; literary language
• GitHub / Stack: code; reasoning in structured, executable form
• Wikipedia: factual, encyclopedic; high signal-to-noise ratio
• ArXiv / PubMed: scientific reasoning; specialised domains
• StackExchange, Reddit: Q&A and conversational patterns

Data Cleaning Pipeline

Raw web data is extremely noisy. Pre-processing steps typically include:

• Language detection: filter to target languages; balance multilingual mix
• Quality filtering: perplexity filtering, heuristic rules (min token count, max repeat ratio)
• Deduplication: exact and near-duplicate removal (MinHash, suffix arrays); prevents memorisation of repeated content
• PII removal: strip emails, phone numbers, credentials
• Toxicity filtering: reduce harmful content in training data

Training Instability

Pre-training long runs encounter instabilities: loss spikes (sudden jumps in training loss), gradient explosions, and catastrophic divergence. Common mitigations:

• Gradient clipping: cap gradient norm to 1.0
• Learning rate warmup: slowly ramp up LR from 0 over ~2000 steps
• Checkpoint averaging: average weights from recent checkpoints
• Loss spike recovery: roll back to last stable checkpoint and skip bad data batch
• Low precision issues: BF16 preferred over FP16 (wider exponent range)

The SFT Data Format

SFT training uses (instruction, response) pairs — demonstrations of the desired behaviour formatted as conversations. Modern chat models use special tokens to delimit turns:

<|system|>
You are a helpful assistant.
<|user|>
Explain gradient descent in simple terms.
<|assistant|>
Gradient descent is an optimisation algorithm
that finds the minimum of a function by
repeatedly taking small steps in the direction
that reduces the function value most steeply...
<|end|>

Only the assistant turns contribute to the loss — the model learns to predict the high-quality response given the instruction.

Dataset Quality vs Quantity

Early SFT research (Alpaca, Dolly) showed that even small high-quality datasets (~52k examples) dramatically improve instruction following. Zhou et al. (2023, LIMA) demonstrated that 1,000 carefully curated examples can produce competitive results — the base model already "knows" the information; SFT just teaches the format.

Key quality signals for SFT data: diverse task coverage, factual accuracy, appropriate length, no contradictions with other examples, and natural conversational flow. Popular open-source datasets: OpenHermes 2.5, Ultrachat, WizardLM/Evol-Instruct, ShareGPT.

Fine-tuning Hyperparameters

SFT uses much lower learning rates than pre-training to avoid catastrophic forgetting of the base model's knowledge:

• Learning rate: 1e-5 to 2e-5 (vs 3e-4 for pre-training)
• Epochs: 1–3 (more risks overfitting on small datasets)
• Cosine decay: LR decays to ~10% of peak by end
• Batch size: 64–256 sequences, often with sequence packing
• Max length: match pre-training context length

Step 1: Reward Model Training

Human labellers compare pairs of model responses to the same prompt and indicate which is better. A separate reward model (typically a smaller transformer) is trained on these preference pairs to predict human preference scores. The reward model learns a scalar measure of response quality that correlates with human judgement across diverse scenarios: helpfulness, accuracy, harmlessness, and style.

# Preference pair example
prompt: "Write a poem about AI"

response_A: "Circuits hum and data streams,
             A mind of silicon dreams..."

response_B: "AI AI AI AI AI AI AI..."

# Human label: A is better
# Reward model learns: score(A) > score(B)

Step 2: PPO Optimisation

The SFT model is used as the starting policy. PPO (Proximal Policy Optimisation) generates completions, scores them with the reward model, and updates the policy to produce higher-scoring outputs. A KL divergence penalty against the original SFT model prevents the policy from deviating too far (reward hacking: outputting nonsensical text that fools the reward model but is worthless to humans).

PPO for LLMs is complex and unstable: it requires loading the policy model, reward model, value model, and reference model simultaneously — 4 copies of model weights.

DPO: Simpler Alternative

Direct Preference Optimisation (Rafailov et al., 2023) reformulates RLHF as a simple supervised learning problem. Given preference pairs (chosen, rejected), DPO directly updates the language model to increase the probability of chosen responses and decrease rejected ones — using a mathematical equivalence between the RLHF objective and a classification loss. No reward model, no PPO, no value function — just two model copies and cross-entropy loss. DPO has become the default alignment method for most open-source models.

LoRA: Low-Rank Adaptation

Hu et al. (2021) observed that the weight updates during fine-tuning have low intrinsic rank — the change ΔW can be well-approximated by a low-rank matrix product BA, where B ∈ ℝ^d×r and A ∈ ℝ^r×k with r ≪ min(d,k).

# Original weight: W (frozen)
# LoRA adds: ΔW = B × A

# Forward pass:
h = W·x + (B·A)·x · (alpha/r)

# Only B and A are trained
# W stays frozen — original knowledge preserved

# r=8: 2×(4096×8) = 65,536 params
# vs full W: 4096×4096 = 16.7M params
# Reduction: 256×

At inference, LoRA can be merged back: W' = W + BA, so there is zero extra latency. Multiple LoRA adapters can be swapped at runtime for different task specialisations.

QLoRA: Quantisation + LoRA

Dettmers et al. (2023) combined LoRA with 4-bit quantisation of the frozen base model weights. The base model is stored in NF4 (Normally distributed Float 4-bit), consuming ~4GB for a 7B model (vs ~28GB in FP16). LoRA adapters remain in full BF16 precision. Double quantisation further reduces memory by quantising the quantisation constants.

Result: fine-tuning a 65B Llama model on a single 48GB GPU — previously requiring 8× A100s. QLoRA democratised fine-tuning for researchers and practitioners without access to multi-GPU infrastructure.