From a GPU incident to a repeatable test

A request gets slower. The GPU might be busy, or the host might have stopped feeding it. A collective can also wait for work on another rank. A utilization chart cannot distinguish all of these cases.

Recent work in lib/nvidia connects the observations around an incident to a trial that can reproduce it. The packages are experimental and under hardware qualification.

Keep the moments before the delay

The flight recorder retains recent Go execution traces and GPU observations in memory. An application can request a snapshot at a workload boundary or after a latency change. Capture budgets limit retained bytes, duration, and concurrent incidents; sampled CPU and memory targets are tracked separately.

Ask a question the trace can answer

Wall time, summed kernel duration, GPU busy time, and host API duration measure different things. Overlapping kernels make their sum especially easy to misread. Diagnostic checks retain those distinctions and require the relevant workload identities and timing evidence before attributing a delay.

Missing scheduler events or uncertain clocks may leave a host-submission diagnosis unresolved. The two-rank NCCL analysis similarly needs matching communicator and operation identities. A missing transport observation stays missing.

Rebuild the conditions

A regression plan records the source, binaries, libraries, and inputs by digest. Preflight checks look for changes in the device, driver, toolkit, clocks, or executable. Missing original artifacts block reconstruction. A changed environment needs an explicitly identified new experiment.

Interleaved baseline and candidate runs reduce ordering effects. Baseline-versus-baseline controls estimate how much variation exists without a code change. The resulting records separate the plan, the execution, and the comparison.

What comes next

The next useful result is a captured workload, reconstructed on the intended GPU, with observations sufficient to explain the delay and a comparison that clears its noise floor. Implementing the workflow is one step toward that result. It does not establish a speedup.

If you have a Go CUDA workload with intermittent stalls, get in touch. The workload, machine, and failure conditions give us a concrete case to review.