Skip to content

THROUGHPUT_ROT — step time is creeping up

Severity: info · Key metrics: step_time_s, generation_time_s, cross-check completion_len_mean

What is happening

Step time in RL fine-tuning is dominated by generation, and generation cost scales with completion length. So when completions grow — length hacking, weakened EOS behavior, a drifting policy that rambles — every training step quietly gets more expensive. A run that starts at 2s/step and rots to 5s/step delivers less than half the training progress per GPU-hour, and nobody notices because throughput isn't on the dashboard.

This detector is severity info by design: throughput rot is almost always a symptom, not a disease. Its value is attribution and cost-accounting — it makes the compute cost of the other pathologies visible, and its evidence says whether completion length grew in parallel (it usually did). Treat it as the "check engine" light that points you at the real fault code.

The genuinely-infrastructure cases (dataloader degradation, memory fragmentation, a checkpoint-writing stall, thermal throttling) do exist — they're what's left when length didn't grow.

What it looks like in the recording

step_time_s with a significant rising trend, late mean ≥ ~1.4× the early mean. The generation/optimization time split (generation_time_s vs optimization_time_s) tells you which half of the step is rotting; length-driven rot lives entirely in generation.

How the detector decides

Fires when the late/early ratio of step_time_s exceeds ratio and the series has a significant Mann-Kendall rising trend — a one-time step up (e.g. a config change mid-run) has no sustained trend and doesn't fire. The finding notes whether completion_len_mean grew ≥1.3× in parallel.

knob default meaning
ratio 1.4 late/early step-time ratio that counts as rot

What to try

  • If length grew in parallel (the common case): this is a symptom — go treat REWARD_HACK_LENGTH or TRUNCATION_STARVE; step time recovers on its own.
  • If not: profile the generation/optimization split, then look at infrastructure — dataloader, checkpoint cadence (visible as checkpoint events aligning with slow steps), memory pressure.

REWARD_HACK_LENGTH and TRUNCATION_STARVE — the usual upstream causes; this finding is their invoice.