Skip to content

REWARD_STD_DEATH — nothing left to learn from

Severity: warn · Key metrics: reward_std, with a which-tail check on reward_mean

What is happening

Policy-gradient methods learn from reward differences, not reward levels. The advantage computation subtracts a baseline (group mean in GRPO, value prediction in PPO), so a batch where every completion scores the same produces zero gradient no matter how high the score is. Reward variance is the fuel; when reward_std → 0 the tank is empty and updates stop mattering, regardless of what reward_mean says.

The diagnostic subtlety — and why the detector explicitly checks which way it died — is that two very different situations produce the identical flat-std curve:

  1. Saturated at the ceiling. reward_mean sits near its maximum: the task, as scored by this reward on this prompt set, is mastered (or hacked to the ceiling — score-maxed isn't the same as solved). Training past this point is free compute converted to heat; the run has outgrown its curriculum.
  2. Died without mastery. reward_mean is mediocre and variance died anyway: the policy collapsed onto one middling behavior everywhere (check entropy), or the reward function itself has a bug — degenerate grading, a broken component, scores pinned by an exception handler.

Same curve, opposite meanings: one says "graduate", the other says "debug".

What it looks like in the recording

reward_std decaying to near zero and holding, zero_variance_group_frac (GRPO) rising toward 1.0 in sympathy, and reward_mean going flat — at its max in case 1, anywhere else in case 2.

How the detector decides

Fires when the late-window mean of reward_std is below death_below and the early mean was at least early_ratio× larger — variance must have died, not merely started small. It then runs the which-tail check: if late reward_mean is within 80% of the run's maximum, the finding says "saturated/hacked to ceiling — task exhausted"; otherwise "variance died without mastery — suspect collapse or a reward bug". The evidence includes reward_at_ceiling so automation can branch on it.

knob default meaning
death_below 0.05 late reward std below this = dead
early_ratio 3.0 early std must have been ≥ this multiple of the late std

What to try

  • Ceiling case: harder prompts (curriculum). Also verify the ceiling is where you think it is — if the reward tops out at 1.0 because of a clamp you forgot, the model may have stopped learning far below actual mastery. And spot-check top-reward samples: "at ceiling" plus garbage text = hacked, not solved.
  • No-mastery case: cross-check entropy_mean (collapse?) and read a batch of samples — identical scores on visibly different-quality completions is a reward bug in plain sight.

ENTROPY_COLLAPSE — the usual culprit in the no-mastery case. DEGENERATE_GROUPS — the per-group early warning; it typically fires many steps before global reward variance flatlines. FORMAT_GAMING — a common route to a hacked ceiling.