The failure-signature catalog

This catalog is the doctor's brain — and it is written to be useful even if you never run rlprobe. Each page explains one known way RL fine-tuning goes wrong: the underlying mechanics, how it shows up in recorded metrics before it shows up in the reward curve, how the detector decides, and what to do about it.

The unifying observation: almost every RL fine-tuning failure has a quantitative signature that appears in recorded data well before it appears in the reward curve. The reward curve is the last thing to know.

The catalog at a glance

Signature	One-line description	Severity
KL_BLOWUP	policy is escaping the reference model	critical
ENTROPY_COLLAPSE	exploration is dead; learning has silently stopped	critical
GRAD_PATHOLOGY	NaN/Inf gradients or violent grad-norm spikes	critical
REWARD_HACK_LENGTH	reward rising because completions get longer, not better	warn
FORMAT_GAMING	one reward component saturated, the rest stagnant	warn
CLIP_SATURATION	most of each update is slamming into the trust region	warn
DEGENERATE_GROUPS	GRPO groups with zero reward variance = zero gradient	warn
ADV_COLLAPSE	advantage std → 0; the learning signal has vanished	warn
VALUE_DIVERGENCE	PPO value head worse than useless; advantages are noise	warn
TRUNCATION_STARVE	rewards computed on cut-off completions	warn
REPETITION_SPIRAL	generated text is looping	warn
REWARD_STD_DEATH	every completion scores the same; nothing left to learn	warn
REF_POLICY_DRIFT_ASYMMETRY	KL concentrated at specific token positions	warn
THROUGHPUT_ROT	step time creeping up (usually a symptom)	info

Critical signatures are auto-halt candidates under the default HaltPolicy; warn-level findings are worth a human look; info is context.

How detectors work

Detectors are deliberately boring: explicit, testable statistics over the recorded series — no ML. The shared toolbox:

• Early/late windows — compare the first ~25% of a run against the last ~20% to measure drift without assuming absolute scales.
• Mann-Kendall trend test — a nonparametric "is this series really rising?" test that doesn't care about the series' distribution.
• Sustained-crossing detection — a threshold only counts when held for N consecutive steps; single-step blips never fire a finding.
• Rolling correlation — for coupling signatures (reward ↔ length).
• Changepoint detection — binary segmentation over z-scored series, used to locate onsets.

Wherever possible thresholds are relative (ratios to the run's own baseline, fractions, trend z-scores) rather than absolute values, so they transfer across model scales better. Every detector is tested both ways: it must fire on its synthetic pathological fixture and must stay silent on healthy runs — false positives destroy trust faster than false negatives.

Findings

Each fired detector emits a structured finding:

{
  "signature_id": "ENTROPY_COLLAPSE",
  "severity": "critical",
  "confidence": 0.9,
  "first_step": 212,
  "explanation": "entropy fell from ~2.31 to ~0.048 — ...",
  "evidence": [{"metric": "entropy_mean", "early_mean": 2.31, "late_mean": 0.048}],
  "suggested_actions": ["add/increase an entropy bonus", "..."],
  "links_to_samples": []
}

first_step is the detector's onset estimate — where the metric first sustainedly crossed its threshold. Detection is threshold-crossing, not clairvoyance: a collapse injected at step 150 might cross the detection threshold at step ~210; the finding points there.

Tuning thresholds

Every detector's thresholds are overridable via doctor.toml (passed as rlprobe doctor <run> --config doctor.toml), one section per lowercase signature id:

[kl_blowup]
ratio = 12.0          # only flag at 12x baseline instead of 8x

[throughput_rot]
enabled = false       # disable a detector entirely

Unknown keys are rejected loudly (a typo must not silently become a no-op). Each signature page lists its knobs and defaults. On Python 3.10 pass the same structure as a dict to diagnose(config=...) — tomllib needs 3.11+.