Skip to content

VALUE_DIVERGENCE — the value head has stopped predicting returns

Severity: warn · Key metrics: value_explained_variance, value_loss · PPO-family only

What is happening

PPO computes advantages against a learned baseline: a value head trained alongside the policy to predict expected return. The baseline's job is variance reduction — advantage = return − V(state) isolates how much better this particular completion was than what the state promised on average. GAE refines this over token positions, but the principle is the same: advantages are only as good as the value function.

Explained variance (EV) is the health meter:

EV = 1 − Var(return − V) / Var(return)

EV ≈ 1: the head predicts returns well. EV ≈ 0: it's useless — advantages degrade into raw returns (high variance, slow learning). EV < 0: the head is worse than predicting the batch mean — it is injecting structured noise into every advantage, and the policy dutifully follows that noise. Updates become random-ish drift that can actively undo learning.

Why value heads diverge: value LR too high (regression overshoots), reward scale too large (huge return targets destabilize the regression), the policy changing so fast that the return distribution shifts under the head every batch, or GAE parameters (γ, λ) mismatched to the episode structure.

What it looks like in the recording

value_explained_variance sinking below zero and staying there, and/or value_loss climbing multiplicatively while everything else looks normal. Policy-side symptoms lag: reward stagnates or wanders after the baseline has already gone bad — the value head is a leading indicator.

How the detector decides

Two independent checks. Primary: EV sustained below ev_floor (default 0.0 — the "worse than useless" line) with the late-window mean also below it. Fallback (when EV isn't recorded): value loss late/early ratio above loss_ratio and a significant Mann-Kendall rising trend in the second half.

knob default meaning
ev_floor 0.0 EV below this = baseline worse than the mean
sustain 10 consecutive steps below the floor
loss_ratio 3.0 late/early value-loss ratio for the fallback check

What to try

  • Lower the value-head learning rate, or give the head its own warmup — let it catch up to the return distribution before the policy starts moving.
  • Revisit GAE parameters (λ, γ) — wrong horizon assumptions make return targets the head structurally can't fit.
  • Check the reward scale — returns in the hundreds make value regression numerically miserable; normalize rewards or scale them down.
  • If the head is far gone, consider re-initializing it and freezing the policy briefly while it re-fits — cheaper than the noise-driven policy drift.

GRPO deliberately deletes this failure mode by replacing the learned baseline with a group mean — at the price of DEGENERATE_GROUPS. The two pages are mirror images: every baseline choice buys one pathology by selling another.