The QA-SFT checkpoint, aligned by reinforcement learning against a reward model, with PPO optimizing the policy toward higher reward while a KL penalty keeps it anchored to the reference.

Trained on Modal L4 (60 PPO iterations). Judged correctness 0.271 [0.238–0.304] — RLAIF held the line where DPO collapsed, landing at parity with SFT (paired win-rate 51%) and far ahead of 125M DPO (0.054). The shared reward model reached 0.983 held-out accuracy.

Architecture

Class	LlamaForCausalLM (from scratch)
Layers	12
Hidden size	768
Attention	12 heads · dim 64 · full MHA
Feed-forward	SwiGLU · inner 3,072
Positional	RoPE · θ 10,000
Norm	RMSNorm · ε 1e-5
Context	1,024 tokens
Vocabulary	16,384 · byte-level BPE
Embeddings	tied input/output

Training

Init from	slm-125m QA-SFT
Method	RLAIF — reward model + PPO
Trainable params	125.8M — all
Training data	500 preference pairs
Training tokens	344K per PPO pass
PPO iterations	60

What this model cost to build

$71.92 total Modal usage

the full lineage. Unlike the 500M and Gemma sites, this base was pretrained from scratch here, so its cost is part of the total rather than someone else's.

Stage	Detail	Cost
125M pretraining (4 legs)	v1 + extended + e2 + e4 · 27.5 h on A100-40GB	$70.14
QA supervised fine-tune	the checkpoint RLAIF starts from · 9.7 min on L4	$0.20
Reward model	Bradley-Terry head on L4	$0.02
PPO alignment	60 iterations · 5.4 min on L4	$0.08
Evaluation (shared)	13 versions on 500 held-out questions, this model's share	$1.48
Total		$71.92

Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.