rlprobe

Flight recorder + doctor for RL fine-tuning runs.

RL fine-tuning (GRPO, PPO, RLOO, DAPO, …) fails in ways that are slow, silent, and expensive. A run can look healthy on its loss curves for hours while the policy is quietly collapsing (entropy → 0), reward-hacking (padding answers for length), or escaping its reference model (KL blowup). By the time a human notices, thousands of GPU-hours are gone — and a wandb dashboard of scalar curves is rarely enough to reconstruct why.

rlprobe adds the two capabilities that are missing:

1. Flight recorder — a low-overhead instrumentation layer that captures a complete, structured, replayable record of a training run: per-step scalar metrics, per-sample forensics (prompts, completions, rewards, advantages), discrete events, and full run configuration. Enough to do a post-mortem without re-running.

2. Doctor — a diagnosis engine that reads the recording (live during training, or post-hoc) and pattern-matches against a catalog of 14 known RL failure signatures, emitting ranked findings: what is wrong, when it started, what evidence supports it, and what to try.

The flight recorder is the black box; the doctor is the NTSB investigator that reads it — except it can also ride along in the cockpit and warn you before the crash.

Architecture

┌─────────────────────────────────────────────────────────────┐
│ TRAINING PROCESS                                            │
│  trainer (TRL GRPOTrainer, or any custom loop)              │
│    └── Recorder (TRL callback, or manual API)               │
│          • async writer thread → near-zero overhead         │
│          • flushes on crash/SIGTERM (black box survives)    │
└──────────────┬──────────────────────────────────────────────┘
               │ writes append-only
               ▼
┌─────────────────────────────────────────────────────────────┐
│ RECORDING (self-contained run directory)                    │
│    manifest.json      config, env, git sha, lib versions    │
│    steps.parquet      per-step scalar metrics               │
│    samples/*.parquet  per-sample forensics (sampled)        │
│    events.jsonl       checkpoints, NaNs, crashes, …         │
└──────────────┬──────────────────────────────────────────────┘
               │ read (live tail or post-hoc)
               ▼
┌─────────────────────────────────────────────────────────────┐
│ DOCTOR (diagnosis engine)                                   │
│  detector catalog → ranked findings                         │
│  CLI report · JSON · HTML report · live alerts · auto-halt  │
└─────────────────────────────────────────────────────────────┘

The recording format is the contract: the recorder knows nothing about diagnosis, the doctor knows nothing about trainers. Either side can be swapped or extended independently, and the doctor works on runs recorded weeks ago.

What it does

• One-line TRL integration — attach(trainer, "runs/my-run") records everything a GRPOTrainer exposes, including per-sample completions and advantages.
• Framework-agnostic manual API — Recorder.log_step / log_samples / log_event for verl, OpenRLHF, or custom loops.
• Live guardian — rlprobe watch tails a growing run and alerts on new findings; opt-in auto-halt stops training on critical, high-confidence findings before more compute burns.
• Post-mortem suite — rlprobe doctor (ranked diagnosis), rlprobe report (self-contained HTML with findings overlaid on charts), rlprobe samples (forensic queries), rlprobe diff (where did run B diverge from run A?).
• Deterministic detectors, no ML magic — thresholds, trend tests, and changepoint statistics over recorded series. Every detector is unit-tested against synthetic pathological runs and healthy runs (false positives destroy trust).
• Crash-survivable, low-overhead recording — async writer, bounded queue that drops rather than blocks training, benchmarked at <2% step-time overhead, and verified at 100k-step scale.

Where to go next

• Quickstart — from install to first diagnosis in minutes, GPU not required.
• Failure-signature catalog — the heart of the project; each page explains the RL mechanics of one failure mode, how it shows up in recorded data, and what to do about it. Useful even if you never run rlprobe.
• Recording format — the on-disk contract, crash safety, and distributed-training semantics.
• API & CLI reference.