Quickstart

Install

pip install rlprobe[trl]      # with the TRL integration
pip install rlprobe           # core only (manual API + doctor)

Note

Until the first PyPI release, install from source: pip install -e ".[trl]" from a checkout.

The doctor has no torch dependency — you can pip install rlprobe on a laptop and diagnose recordings copied from a training box.

Record a TRL run (3 lines)

from rlprobe.integrations.trl import attach

trainer = GRPOTrainer(...)          # your existing setup
attach(trainer, "runs/my-run")      # ← the integration
trainer.train()

That's it. The callback records every scalar TRL logs (reward, KL, entropy, clip ratio, per-component rewards, …), plus per-sample forensics — prompts, completions, rewards, advantages, GRPO group ids — for the highest/lowest-reward samples each step.

Diagnose it

rlprobe doctor runs/my-run

CRITICAL ENTROPY_COLLAPSE | onset ~step 212 | confidence 90%
entropy fell from ~2.31 to ~0.048 — the policy is (near-)deterministic and
exploration is dead, while reward has plateaued (classic collapse pair)

try:
  • add/increase an entropy bonus
  • raise the sampling temperature during rollouts
  • check whether the reward is saturated (nothing left to explore for)

Exit code 1 on critical findings, so it works as a CI gate. Add --json for machine-readable findings.

Try it without a GPU

The synthetic run generator fabricates realistic healthy or pathological recordings — the same fixtures the detector test suite uses:

rlprobe synth runs/demo --pathology entropy_collapse --n-steps 500
rlprobe doctor runs/demo
rlprobe report runs/demo          # self-contained HTML post-mortem

Known pathologies: every signature in the catalog, e.g. kl_blowup, reward_hack_length, degenerate_groups, grad_pathology.

Watch a run live

rlprobe watch runs/my-run --poll 3 --webhook https://hooks.slack.com/...

The watcher tails the growing recording and reports each new finding once. For the in-process version with automatic halting:

attach(trainer, "runs/my-run", halt=True)   # stop on critical, high-confidence findings

A halted run logs an auto_halt event and sets callback.halted_by to the finding that pulled the trigger. The default policy is conservative — critical severity at ≥70% confidence — because a warn-level hunch must never burn a checkpoint. See HaltPolicy to tune it.

Post-mortem workflow

Everything works from the CLI, no notebook required:

Question	Command
What went wrong, and when?	rlprobe doctor runs/my-run
Show me everything, shareable	rlprobe report runs/my-run → single HTML file
What was the model generating around step 2100?	rlprobe show runs/my-run --step 2100
Which completions drove the reward spike?	rlprobe samples runs/my-run --sort advantage --around 2100
Where does run B diverge from run A?	rlprobe diff runs/A runs/B
What runs do I have?	rlprobe ls runs/

Custom training loop (manual API)

The TRL adapter is a thin layer over the framework-agnostic core — any loop can use it directly:

from rlprobe import Recorder

with Recorder("runs/custom", algorithm="grpo", trainer_config=cfg) as rec:
    for step in range(n_steps):
        metrics, samples = train_step(...)
        rec.log_step(step, reward_mean=..., kl_mean=..., entropy_mean=..., **metrics)
        rec.log_samples(step, samples)          # dicts with completion_text, reward_total, ...
        if step % 500 == 0:
            rec.log_event("checkpoint", step=step, message=f"ckpt-{step}")

log_step accepts any numeric keyword — unknown names land in extra and are recorded (and charted) like first-class metrics. See the API reference for the full surface and the recording format for what each field means.