Skip to content

CLIP_SATURATION — updates keep slamming into the trust region

Severity: warn · Key metrics: clip_fraction, ratio_p95

What is happening

PPO and GRPO use a clipped surrogate objective. For each token they compute the probability ratio r = π_new(token) / π_old(token) between the current policy and the policy that generated the rollout, and clip it to [1-ε, 1+ε] (ε ≈ 0.2). Outside that band, the gradient is zeroed: the clip is a trust region that says "don't move further than this on data generated by a policy you no longer are."

A healthy run clips roughly 5–15% of tokens — the trust region should bite occasionally; that's it doing its job. Sustained clipping of 30%+ means every update wants to move far outside the trust region and most of the computed gradient is being thrown away. What survives the clip is a biased sliver of the intended update, so learning becomes slow and distorted at the same time — you pay full compute for a fraction of a gradient you can't fully trust.

Causes: learning rate too high (each step overshoots the region), too many optimizer epochs per rollout batch (by the later epochs, π_new has drifted far from the π_old that generated the data), or stale rollouts in async setups (same mechanism — the data is from a policy too many updates old).

What it looks like in the recording

clip_fraction stepping up and holding above ~0.3 for tens of steps. ratio_p95 drifting well above 1+ε corroborates it and reveals whether the pressure is asymmetric (policy consistently pushing probabilities up on its own tokens).

How the detector decides

Fires when clip_fraction stays above threshold for sustain consecutive steps and the late-window mean is also above threshold — a transient burst after an LR schedule change doesn't fire.

knob default meaning
threshold 0.3 clip fraction that counts as saturated
sustain 20 consecutive steps above threshold

What to try

  • Lower the learning rate — the most direct way to make proposed updates fit inside the trust region.
  • Fewer optimizer epochs per rollout batch — fresher data keeps π_old close to π_new. This trades sample efficiency for update quality, which is usually the right trade once clipping saturates.
  • Check ratio_p95 for asymmetric drift — a one-sided ratio distribution points at systematic probability inflation rather than noisy overshoot.

KL_BLOWUP — the same "moving too fast" disease measured against the frozen reference instead of the rollout policy; they frequently co-occur, and the LR remedy is shared.