Skip to content

REWARD_HACK_LENGTH — reward is rising because completions get longer

Severity: warn · Key metrics: completion_len_mean, reward_mean, linked samples

What is happening

The most common reward hack in practice. Almost any imperfect reward function has a soft spot for length: more tokens mean more chances to hit a keyword, more partial credit for "covering" the answer, more matches for a lenient grader, higher scores from an LLM judge that conflates thorough with verbose. If length pays at all, it is usually the cheapest gradient direction available — padding is much easier to learn than actually being right — so the optimizer takes it.

The result: reward climbs, dashboards look great, and the true quality of outputs is flat or worse. Left alone, length growth also drags in secondary failures: rising generation cost (THROUGHPUT_ROT) and completions slamming into the token cap (TRUNCATION_STARVE).

This is a specific instance of Goodhart's law — the metric (reward) stopped measuring what you meant (quality) the moment the policy found the seam between them.

What it looks like in the recording

completion_len_mean growing steadily (the detector requires ≥1.4× growth), with reward_mean tracking it closely — sustained rolling correlation above ~0.75. In healthy runs reward and length move independently; sustained high correlation while length grows is the tell.

The finding links the longest high-reward samples from the run — read them. Padding, filler phrases, restated questions, and boilerplate chains-of-thought are usually visible to the naked eye, and per-sample forensics is exactly why the recorder captures completion text.

How the detector decides

Two conditions must hold: late-window mean length ≥ len_growth× the early mean, and the rolling correlation between reward and length staying above corr_threshold for sustain consecutive steps. Correlation alone never fires — a run whose lengths are stable but noisily correlated with reward is not hacking.

knob default meaning
len_growth 1.4 minimum late/early length ratio
corr_threshold 0.75 rolling corr(reward, length) that counts as coupled
corr_window 25 rolling-correlation window (steps)
sustain 15 consecutive steps above the correlation threshold

What to try

  • Add an explicit length penalty, or normalize reward by length — make padding cost what it earns.
  • Read the linked samples first — confirm the mechanism before changing the reward. If long answers are genuinely better for your task, this finding is a false alarm and you should raise the thresholds instead.
  • Fix the reward's soft spot at the source: exact-match graders instead of containment checks; judge prompts that explicitly score conciseness.
  • Tighter max_new_tokens as a stopgap — caps the damage but grades truncated text (see TRUNCATION_STARVE); treat it as a tourniquet, not a fix.

FORMAT_GAMING is the same exploit against a component of a composite reward. TRUNCATION_STARVE and THROUGHPUT_ROT are its downstream symptoms.