Skip to content

REF_POLICY_DRIFT_ASYMMETRY — KL concentrated at specific token positions

Severity: warn · Key metrics: kl_positional (per-position KL profile)

What is happening

Some drift from the reference model is the point of RL fine-tuning — the policy is supposed to change. Healthy drift is roughly uniform across token positions: the policy got a bit better everywhere. Drift concentrated at specific positions — classically the first few tokens of the completion — is a different animal, and it usually isn't learning at all.

The mechanism: the KL penalty compares log π_policy(token) against log π_ref(token) position by position, assuming both models saw the same context. If the policy and reference are fed even slightly different prompt encodings — a chat-template difference, a doubled or missing BOS token, a system prompt present on one side only, different whitespace around role markers — then at the positions right after the mismatch the two models are answering different questions, and the measured KL there is huge no matter what the policy learned.

The damage is quiet and systematic: the KL budget is being spent on a formatting artifact, so the effective penalty on real behavioral drift is weaker than configured everywhere else — and the optimizer is actively pushed to contort the first tokens to appease a miscomputed penalty. Runs like this often "work" while being subtly worse than they should be, which is exactly the kind of failure that never shows up on a reward curve.

This is why the recorder keeps a positional KL profile (KL bucketed by token position) rather than only the scalar mean — the scalar hides exactly this shape.

What it looks like in the recording

The kl_positional matrix showing one bucket (typically bucket 0 — the first tokens) carrying a large share of the total KL mass in the late window, while other buckets stay near baseline. kl_mean may look merely "somewhat elevated".

How the detector decides

Averages the positional profile over the late window and fires when a single bucket's share of total KL mass exceeds share_threshold — with guards requiring enough buckets to be meaningful and enough total KL to be worth attributing.

knob default meaning
share_threshold 0.45 one bucket owning more than this share of KL mass fires
min_kl 0.05 minimum total KL mass (don't attribute noise)
min_buckets 4 minimum profile resolution to judge concentration

What to try

  • Diff the exact tokenized inputs fed to policy and reference for one sample — not the strings, the token ids. This finds the mismatch in minutes and is the fix in most cases.
  • Audit chat template / BOS / system-prompt handling on both paths, especially after upgrading the training framework or tokenizer — these paths drift apart across library versions notoriously easily.
  • If the inputs truly match, the concentration is real behavior (e.g. the policy learned a new opening move); judge it on the merits and consider whether the KL coefficient needs adjusting.

KL_BLOWUP — elevated KL everywhere, the genuine escape. Check this signature before reacting to that one: a template mismatch inflates scalar KL and can make a healthy run look like it's escaping.