Skip to content

ENTROPY_COLLAPSE — exploration is dead

Severity: critical (auto-halt candidate) · Key metrics: entropy_mean, cross-checked against reward_mean

What is happening

Policy entropy measures how much probability mass the policy spreads over alternatives at each token — in plain terms, how much it still explores. RL learns from contrast: GRPO compares completions within a group, PPO compares against a baseline; either way, the gradient only exists where sampled outcomes differ.

A near-deterministic policy (entropy ≈ 0) generates essentially the same completion for a prompt every time. Same completions → same rewards → zero advantage → zero gradient. Learning doesn't fail loudly; it silently stops, and compute keeps burning. The characteristic pair on the dashboard is entropy sliding to zero while reward plateaus — the plateau isn't convergence, it's the policy having nothing left to compare.

Why it happens:

  • No entropy bonus — nothing in the objective pays for keeping options open, and reward maximization naturally sharpens the distribution.
  • Sampling temperature too low during rollouts — exploration was throttled at the source.
  • LR too high early — the policy over-commits to the first behavior that scored well before it saw alternatives.
  • Reward saturation — if everything already scores maximum, sharpening onto the current behavior is optimal; the collapse is a symptom (see REWARD_STD_DEATH).

What it looks like in the recording

entropy_mean decaying — often roughly exponentially — and then holding near zero, while reward_mean flattens. For reference: a uniform random policy over a 152k-token vocabulary has entropy ≈ ln(152k) ≈ 11.9; a healthy mid-training policy typically sits in low single digits; below ~0.25 the policy is effectively deterministic.

How the detector decides

Fires when the late-window mean of entropy_mean is below collapse_below and the early-window mean was above healthy_early — a policy that started sharp is a configuration choice, not a collapse. As supporting evidence it runs a trend test on late reward: if reward has plateaued too (|z| < 1.96, i.e. no significant trend), confidence rises from 0.75 to 0.9 — the classic pair is stronger evidence than entropy alone.

knob default meaning
collapse_below 0.25 late entropy below this = collapsed
healthy_early 1.0 early entropy must have been above this
sustain 3 consecutive steps below threshold for the onset estimate

What to try

  • Add or increase an entropy bonus — pay the policy directly for keeping options open.
  • Raise the rollout sampling temperature — restores exploration at generation time without touching the objective.
  • Check whether reward is saturated — if there is genuinely nothing left to learn on this prompt set, the fix is harder prompts, not more entropy.
  • If caught early, lowering LR can slow the sharpening; a fully collapsed policy usually needs a checkpoint restore — the alternatives it would need to explore no longer get sampled at all.

REWARD_STD_DEATH distinguishes "converged at the ceiling" from "collapsed without mastery" — read it together with this one. REPETITION_SPIRAL is often the leading edge: looping text is what a low-entropy policy sounds like.