Skip to content

ADV_COLLAPSE — the learning signal has vanished

Severity: warn · Key metrics: adv_std, adv_zero_frac

What is happening

Advantages are the actual learning signal of policy-gradient RL. The gradient for a sample is (roughly) advantage × ∇ log π(completion) — the advantage is the scalar that says how hard, and in which direction, to push the policy toward or away from that completion. Rewards are raw material; advantages are what the optimizer eats.

If the standard deviation of advantages across the batch goes to ~0, every sample is telling the policy "you were exactly as good as expected" — and the gradient vanishes globally. The run keeps burning compute, the loss keeps printing numbers, and nothing is being taught. Reward curves can look perfectly stable throughout, because this failure lives one derivative away from what dashboards usually plot.

Distinct causes with the same curve:

  • Reward normalization/scale bug — e.g. a whitening layer or reward-scaling wrapper mis-applied so differences are squashed before advantages are computed.
  • Total reward saturation — everyone at the ceiling; no differences left to measure (the honest version).
  • Degenerate GRPO groups everywhere — per-group zero variance aggregating into global zero variance (see DEGENERATE_GROUPS).

What it looks like in the recording

adv_std sliding from a healthy ~1.0 (it is ~1 by construction when group/batch normalization is working and rewards vary) down to near zero and holding there; adv_zero_frac rising in parallel.

How the detector decides

Fires when the late-window mean of adv_std is below collapse_below while the early mean was above healthy_early — the run must have had signal and lost it; a run that never had usable advantages is a setup problem the healthy-early guard refuses to misdiagnose.

knob default meaning
collapse_below 0.1 late adv_std below this = collapsed
healthy_early 0.5 early adv_std must have been above this
sustain 5 consecutive steps below threshold for the onset estimate

What to try

  • Audit reward scaling/normalization first — this is the classic silent bug: the pipeline computes rewards correctly and then flattens them on the way to the advantage computation.
  • Check for saturation — if reward_mean sits at its ceiling, the fix is harder prompts, not advantage plumbing.
  • Cross-check zero_variance_group_frac and reward_std — they locate the collapse: per-group (curriculum problem), global reward (task exhausted), or advantages only (normalization bug between the two).

DEGENERATE_GROUPS and REWARD_STD_DEATH are the upstream views; if all three fire together, start from reward variance. If only this one fires, the bug is almost certainly in the reward→advantage plumbing.