Skip to content

GRAD_PATHOLOGY — NaN/Inf gradients or violent grad-norm spikes

Severity: critical (auto-halt candidate) · Key metrics: nan_count, inf_count, grad_norm

What is happening

This is the closest thing the catalog has to a smoke alarm — not a subtle optimization dynamic but numerical damage in progress.

NaN/Inf gradients poison weights permanently. One NaN in the gradient becomes NaN in the touched weights after the optimizer step, then NaN activations, then NaN everywhere. Optimizer momentum states can carry the damage even past a lucky-looking step or two. A single bad update can zero the value of an entire run; everything trained after it is suspect.

Grad-norm spikes (orders of magnitude above the typical norm) are the same disease in its early, recoverable stage: numerical instability from fp16 overflow, a pathological batch (one absurd reward, one malformed sample), or an LR that turns a sharp loss-surface region into a catapult. Gradient clipping usually absorbs isolated spikes — recurring spikes mean the source is systematic and clipping is the only thing between you and the NaN case.

The classic precision trap: fp16 has a maximum value of 65504 — attention logits and loss-scaling mishaps overflow it routinely. bf16 keeps fp32's exponent range (trading mantissa precision) and eliminates most of this failure class outright.

What it looks like in the recording

nan_count/inf_count nonzero at any step (the recorder also writes a nan_detected event, so it survives even if the run dies immediately after), or grad_norm showing repeated spikes far above its own median. The finding pinpoints the first bad step — that, minus one checkpoint, is your restore point.

How the detector decides

Two independent checks. NaN/Inf: any nonzero count fires at 0.99 confidence — there is no innocent explanation. Spikes: at least min_spikes steps where grad_norm exceeds spike_ratio× the run's own median (relative to the run's scale, so it transfers across model sizes).

knob default meaning
spike_ratio 10.0 multiple of the median grad norm that counts as a spike
min_spikes 3 spikes required to fire (one clipped spike is survivable noise)

What to try

  • fp16 → bf16 if your hardware allows it; otherwise verify loss scaling is actually enabled and functioning.
  • Restore from a checkpoint before the first NaN — do not keep training on possibly-poisoned weights; the first bad step in the finding tells you where.
  • Tighten gradient clipping and/or lower LR for the spike case.
  • Inspect the spiking steps' samples (rlprobe show <run> --step N) — a single outlier reward or malformed prompt in the batch is a common, fixable source.

Usually a run-killer on its own, but check what preceded the first spike: KL_BLOWUP or exploding completion lengths sometimes supply the pathological batches that trip it.