Skip to content

FORMAT_GAMING — one reward component saturated, the rest stagnant

Severity: warn · Key metrics: per-component series in reward_components

What is happening

Composite rewards — total = w₁·correctness + w₂·format + … — are standard practice: you want the model to be right and to answer in the expected structure. But the optimizer doesn't care about your intent, only about which term buys the most reward per unit of effort. Format-ish components (uses the right tags, valid JSON, contains a \boxed{}) are usually trivial to max out compared to being correct.

So the model farms the easy term: the format component climbs to its ceiling and pins there, total reward goes up, and the components you actually care about stop moving. The total-reward curve looks like progress; decomposed, it's the model collecting its participation trophy over and over.

This is why the recorder captures per-component reward series, not just the total — the signature is invisible in the aggregate.

What it looks like in the recording

One component's series climbs and then plateaus pinned at its maximum (a sustained stretch within a couple percent of its ceiling), while every other component shows no significant rising trend over the second half of the run.

How the detector decides

For each component: it must have actually climbed to its ceiling (rise ≥ min_rise_frac of its value range — a constant component is a configuration, not a game) and then stayed within ceiling_tol of the ceiling for at least plateau_frac of the late window. If, additionally, no other component has a significant Mann-Kendall rising trend in the second half, the finding fires. Checking the second half only matters: components that grew healthily early would otherwise mask the stagnation.

knob default meaning
ceiling_tol 0.02 within this fraction of the value range counts as "at ceiling"
plateau_frac 0.8 fraction of the late window that must sit at the ceiling
min_rise_frac 0.3 the component must have risen ≥ this fraction of its range

What to try

  • Rebalance the weights — down-weight the saturated term; it has nothing left to teach.
  • Binary-gate the easy component instead of scaling it: format compliance as a multiplier (0 or 1) on the substantive reward removes it as an independent income stream.
  • Make the saturated component harder if it's supposed to keep teaching (stricter schema, exact formatting).
  • Verify the stagnant components still vary on current outputs — if correctness is 0 on every sample, its gradient is zero for a different reason (see DEGENERATE_GROUPS) and rebalancing won't help.

REWARD_HACK_LENGTH — the same "cheapest term wins" dynamic expressed through length instead of a component. REWARD_STD_DEATH — where farming ends up once the easy term is exhausted.