How Meta uses GPU spares to keep Prometheus training

Meta reserves eight of every 72 GPUs in a Prometheus NVL72 domain as in-rack spares, leaving 64 for an active training slice. The plan pairs that spare capacity with fault isolation and checkpoint recovery so large training jobs can continue when hardware or other components interrupt work.

Prometheus is Meta’s AI cluster in New Albany, Ohio. NeoTeo previously covered its network and construction, which provide context for the infrastructure behind this reliability strategy.

Why each NVL72 domain has spare GPUs

Meta describes each NVL72 domain as a 72-GPU unit, with 64 GPUs assigned to a training slice and eight held in reserve. That is about 11.1% of the domain’s GPUs. The eight are reserved capacity, not a count of failed devices; a hardware fault may prompt the system to use one.

The larger domain also concentrates more GPUs within a tightly coupled NVLink fabric. Meta says the potential fault blast radius for NVL72 is nine times that of the earlier H100 NVL8 domain, which grouped eight GPUs. The in-rack spares are one part of Meta’s response to that greater potential impact. It also holds roughly 2% additional racks for rarer faults that can take an entire rack or pod out of service.

An interruption is broader than a failed GPU: Meta defines it as a fault that forces a training job to pause and recover. The causes can include GPU silicon, networking, storage or thermal issues, so spare GPUs address only one part of the recovery plan.

How Prometheus recovers from faults

When a fault is isolated to one data-parallel replica, Meta says that replica can be taken offline while the others continue training. The affected slice can then restart on spare hosts and resynchronize. This limits the disruption to the portion of work affected by that fault.

For a fault with a wider impact, the job can resume from its last checkpoint, a saved copy of the model’s training state. Meta says its distributed checkpointing saves model state, including gradients and optimizer data, asynchronously to multi-tier storage. That gives the job a recovery point rather than requiring it to restart from the beginning.

Meta’s example for a run with 32,768 active GPUs uses 523 NVL72 racks and lists 37,656 GPUs in total. The separate reserve of roughly 2% additional racks is intended for less common, whole-rack failures.

What Meta reports about reliability

In an engineering post published October 19, 2026, Meta said interruptions at Prometheus peaked at nearly 1,000 events per day per 8,000 GPUs during the cluster’s initial phase in mid-2025. Meta said about a year of engineering changes reduced interruptions by 99%.

For Muse Wonder training runs, Meta reported a wall-clock mean time between interruptions (MTBI) of about 47,000 GPU-hours, later rising to more than 54,000 GPU-hours after fleet changes. Its stated target is about 100,000 GPU-hours MTBI, equivalent in Meta’s account to two interruptions per day per 8,000 GPUs. That figure remains the target, separate from the reported results.

Meta also reported goodput above 85% for Muse Wonder pretraining runs and above 90% for its reinforcement-learning runs. Those figures apply to the specified workloads.

Meta described Prometheus as having hundreds of thousands of GPUs and scaling toward roughly one million at full scale. Its October 19 post identified NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 as deployed platforms, with NVIDIA Vera Rubin NVL72 planned for a later expansion.