Diagnose Distributed AI Workload Failures with NVIDIA Attribution

In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began.

In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence, inspect the diagnosis, and explore how attribution rules identify failure patterns.

NVIDIA Attribution correlates workload-scoped logs, metrics, telemetry, and scheduler state and returns a structured diagnosis: outcome, root cause, affected nodes, supporting symptoms, and evidence trail. Workload-centric means analyzing failures in the context of a workload rather than as isolated node events. Deterministic rules make the results reproducible and customizable, while structured output enables integration with existing health and recovery systems.

NVIDIA health and resilience software helps teams keep AI workloads running. NVIDIA Attribution explains why a workload failed and provides evidence to guide the next action. It complements readiness checks, continuous health monitoring, and the systems that repair infrastructure and restart workloads.

NVIDIA Attribution, the workload-centric failure-attribution system used across NVIDIA AI infrastructure, is open source.

Used in production across NVIDIA AI infrastructure for years, NVIDIA Attribution analyzes millions of workloads each week across hundreds of thousands of GPUs. In one large RDMA-backed cluster, it correlated port errors, connection resets, and scheduler events to identify patterns around specific host and switch ports, helping teams target fabric repairs and congestion tuning.

It includes two deployable components: the NVIDIA Attribution Service and the NVIDIA Attribution Blueprint; both components utilize a core attribution engine. The Service makes attribution available through a RESTAPI; the Blueprint connects cluster schedulers and telemetry sources to the Service and writes the result.

Turn workload evidence into one diagnosis

NVIDIA Attribution Service analyzes one workload attempt at a time. The Blueprint gathers the workload context and evidence, the Service extracts and correlates events, and the Blueprint returns a structured result.

The path has four stages:

Scope the workload. The Blueprint identifies the workload attempt from scheduler identifiers or from an explicit execution window and node allocation. Collect observations. Log and metric connectors produce workload-scoped Observation records from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, node and network metrics. Extract and correlate events. The Service invokes the attribution engine to identify significant events such as GPU XIDs, NCCL failures, scheduler transitions, and metric conditions. Deterministic rules evaluate their timing and interaction, separate downstream symptoms from the event that caused the failure, and identify the root cause. Return a structured result. The Blueprint combines the verdict with the workload outcome (terminal_state), attributed domain (sub_domain), symptoms, affected nodes, and a record of collector coverage and errors.

From applogs to attribution

What follows is a worked example of extracting evidence, interpreting events, and preserving the symptoms that explain a distributed workload failure.

Consider a multi-rank training workload in which one GPU encounters a hard fault e.g: XID 149 and NCCL errors follow across other ranks. NVIDIA Attribution correlates events across nodes, identifies the XID as the root cause, retains the NCCL failures in the symptom chain, and names the responsible node. The workload is reported as a hardware failure rather than a set of independent communication errors.

A failed collective can leave error reports on several nodes. Those reports tell us where the workload noticed a problem. To explain the failure, NVIDIA Attribution combines them with evidence from other sources and evaluates the symptoms together.

In this synthetic workload, application logs contain three different NCCL errors. System logs on node-02 contain XID 149 with error words that decode to NVLink signal integrity issue. We will follow that evidence through the rules pipeline.

The Extraction phase is the first to run; it runs on the raw input logs and produces structured events to be processed by later stages.

Decode NVLink evidence during filtering

Filter rules read and write the event stream. The EARLY-stage HandleXidBurstEvents rule reads the XID burst evidence, decodes the error words, and emits an attribution event for signal integrity issue on node-02. The three NCCL events remain unchanged during this step.

Figure 2, above, The figure abbreviates the decoded symptom as XID-149.4 signal integrity issue. The filter also emits parser information for inspection.

Refine the symptoms into one attribution

The framework materializes events into per-node symptoms, runs per-node scrubbing, and aggregates the surviving symptoms into a job-wide set. In this example, all four candidates reach FINAL refinement.

Xid149AndNvlinkFiltering retains the signal integrity symptom and demotes the other node symptoms. Gb200PriorityRootCause reduces the active job-wide symptom set. Outcome mapping then produces the hardware-failure attribution.

The attribution surfaces node-02 and signal integrity issue. The NCCL reports remain useful evidence: demotion preserves their original sources as “interesting,” so they can still be inspected.

When multiple causes fit the evidence, sub_domain is ambiguous. Quality of attribution is dependent on the quality and completeness of the underlying telemetry. The diagnosis records that distinction, correctly reflecting uncertainty instead of converting incomplete evidence into a definitive diagnosis.

Service and Blueprint architecture

The attribution engine contains the rules engine and core attribution logic. The Attribution Service exposes that logic through an HTTP API, and the Blueprint handles cluster-specific integration. This separation lets teams apply the same attribution logic across clusters while adapting scheduler, telemetry, and output connectors to each environment.

Attribution Service: A self-hosted HTTP wrapper around the core attribution engine that receives normalized observations and returns one verdict for each workload.

Attribution Blueprint: A Python reference orchestrator that discovers workloads, resolves the ExecutionContext, runs collectors, calls the Service, enriches the verdict, writes the AttributionBlueprintResult, and supplies a read-only dashboard. This is intended to be a reference implementation for practitioners to customize the solution to their environment.

The Blueprint ships with:

Execution-context connectors for Slurm, Kueue, and static inputs Log and metric connectors for Loki, local files, Kubernetes pod logs, and PromQL metrics Output connectors for stdout and SQLite

Blueprints can run near each cluster’s data sources, with either a dedicated Attribution Service for each Blueprint or a shared Service for multiple Blueprints. There is flexibility on how you want to customize your deployment, e.g: one to one or one to many. Downstream systems consume terminal_state and sub_domain instead of parsing product-specific logs.

Extending Attribution to your infrastructure

Open sourcing NVIDIA Attribution enables infrastructure teams to encode a failure pattern once and apply it across workloads and clusters. Developers can add rules for new failure modes, implement connectors for their schedulers and telemetry backends, and route AttributionBlueprintResult records into incident, recovery, and fleet-management systems. Because extensions use the same ExecutionContext, Observation, and result contracts, new integrations feed the same diagnosis pipeline.

Walkthrough

The earlier example illustrates how a GPU fault can lead to NCCL errors across ranks. In this walkthrough, you’ll use NVIDIA Attribution to analyze NVLink error evidence from a recorded training workload. You’ll run the included sample, inspect the diagnosis and source evidence, and examine the rule that narrows the reported symptoms. You’ll then use the NVIDIA recovery guidance to identify the next action for the affected node.

The walkthrough uses two components:

The Attribution Service runs the analysis rules. The Attribution Blueprint reads the workload context and logs, calls the Service, and stores the result for inspection in a local dashboard.

The sample evidence ships with the repository, so you can follow the analysis before connecting a live cluster.

Run the sample workflow

The NVIDIA Attribution quickstart builds the Attribution Service and dashboard in Docker, then uses the Blueprint to process sample workload evidence committed to the repository. You need Docker and the uv package manager.

Clone the repository and run the sample workflow:

Open http://127.0.0.1:8080 to inspect each workload’s diagnosis, affected nodes, supporting symptoms, and evidence trail.

The rules engine uses checked-in declarative and Python rules. This production rule recognizes an invalid Pyxis argument in an application log and includes a co-located test case:

At startup, the Service loads the bundled rule catalog and evaluates the inline cases. A rule change is committed and shipped with the Service; rules are not uploaded dynamically through the HTTP API. The repository’s rule-authoring guide explains how to select the appropriate rule form, register new symptoms, add tests, and verify the complete catalog.

Get started

Clone the NVIDIA Attribution repository and follow the quickstart to run the sample and inspect its diagnoses. To use Attribution with your own infrastructure, see the Blueprint configuration reference for scheduler and telemetry connectors, and learn how to add an extractor rule for new evidence patterns.

Acknowledgements

Aaron Blakeman, Ajay Hotchandani, Alex Shaposhnikov, Ashwin Poojary, Ben Gueret, Chunyang Sheng, Dorrin Poorkay, Gargi Prasad, Felix Abecassis, Grace Tan, Jason Sewall, Mayuri Gupta, Pasha Shamis, Pavan Kumar Gondhi, Piero Orderique, Rahul Kandu, Rakesh Konda and Sirshak Das contributed to NVIDIA Attribution and to this post.