Skip to content

KL_BLOWUP — the policy is escaping the reference model

Severity: critical (auto-halt candidate) · Key metrics: kl_mean, kl_max

What is happening

RLHF-style objectives maximize reward minus a KL penalty to a frozen reference model (usually the SFT checkpoint). That penalty is not a regularization nicety — it is what tethers the policy to its pretrained competence. The reward function only judges reliably on text that looks like what it was built for; the KL term is what keeps the policy inside that region.

When KL grows super-linearly, the optimizer has found that sprinting away from the reference is the cheapest way to farm reward. Scores may keep climbing while the actual text degrades into off-distribution output the reward function can no longer judge — reward hacking with the safety rope cut. Past a certain drift the run is usually unrecoverable without restoring a checkpoint, which is why this is a critical, halt-worthy signature.

Common causes, roughly in order of frequency:

  • KL coefficient (β) too low or accidentally zero — the tether isn't attached.
  • Learning rate too high — each update overshoots; the penalty can't keep up.
  • Reference-model placement bug — the "frozen" reference is being updated, or the wrong weights were loaded, so the penalty measures drift against a moving target.

What it looks like in the recording

kl_mean rising with accelerating slope, typically over tens of steps; kl_max rising even faster (the drift is usually worst on some tokens first). Reward often still looks good — that's the trap.

How the detector decides

Fires when the late-window mean of kl_mean exceeds ratio× the early-window baseline (with an absolute floor so a near-zero baseline can't make 8× trivial), and the second half of the series shows a statistically significant rising trend (Mann-Kendall, p < 0.05). Requiring both means a one-off spike or a high-but-stable KL plateau doesn't fire. Onset = first sustained crossing of the threshold.

knob default meaning
ratio 8.0 late mean must exceed this multiple of the early baseline
abs_floor 0.5 minimum absolute late KL to fire (guards near-zero baselines)
sustain 3 consecutive steps above threshold for the onset estimate

What to try

  • Raise β (the KL coefficient) — or re-enable it if it's set to 0. This is the direct fix: make escaping more expensive than improving.
  • Lower the learning rate — smaller steps give the penalty a chance to pull back.
  • Verify the reference model — confirm it is frozen, on the right device, and the weights you think it is. Placement bugs produce exactly this curve and no hyperparameter will fix them.
  • Restore from a checkpoint before the onset step; heavily drifted weights rarely come back.

REF_POLICY_DRIFT_ASYMMETRY — when KL is elevated but concentrated at specific token positions, suspect a template mismatch rather than a genuine escape. ENTROPY_COLLAPSE often co-occurs late: a policy that escaped the reference tends to collapse onto whatever hack it found.