Skip to content

DEGENERATE_GROUPS — GRPO groups with zero reward variance

Severity: warn · Key metrics: zero_variance_group_frac, group_reward_std_mean, adv_std · GRPO-family only

What is happening

GRPO replaces PPO's learned value baseline with a group baseline: sample G completions per prompt, and compute each one's advantage as

advantage = (reward − group mean) / group std

The elegance is also the failure mode. If every completion in a group scores the same — the prompt is too easy (all correct) or too hard (all wrong) — then reward minus group mean is zero for every member. The whole group contributes exactly nothing to the gradient. You paid for G generations, G reward evaluations, and a backward pass, and learned nothing from that prompt.

Some zero-variance groups are normal (5–10%). A rising fraction means the prompt difficulty no longer matches the policy: as the model improves, more prompts drop into "all correct"; a too-hard dataset starts at "all wrong". Binary rewards make this dramatically worse than continuous ones — there are only two scores for a group to agree on. At 40%+ zero-variance groups, most of your batch is dead weight and the effective batch size has quietly shrunk to a fraction of what you're paying for.

What it looks like in the recording

zero_variance_group_frac climbing and holding above ~0.4, with group_reward_std_mean sinking in parallel. adv_std shrinks as a downstream consequence — see ADV_COLLAPSE, which is the same disease measured one stage later.

How the detector decides

Fires when zero_variance_group_frac stays above threshold for sustain consecutive steps and the late-window mean is above threshold as well.

knob default meaning
threshold 0.4 fraction of zero-variance groups that counts as degenerate
sustain 15 consecutive steps above threshold

What to try

  • Rebalance prompt difficulty — curriculum in the literal sense: mix in harder prompts when groups are all-correct, easier ones when all-wrong. The useful prompts are the ones the policy gets sometimes right.
  • Increase group size — more completions per prompt means more chances for a disagreement; cost scales linearly, so this buys signal with compute.
  • Raise the sampling temperature — more diverse completions disagree more often.
  • Soften the reward — partial credit (per-step correctness, graded formats) gives groups more scores to disagree over than a 0/1 grader allows.

ADV_COLLAPSE — the global version of the same starvation. REWARD_STD_DEATH — when the whole batch, not just groups, has stopped disagreeing.