Checkpoint Storage That Doesn't Stall Training
Meta's Llama 3 run hit an interruption every ~3 hours on 16K GPUs. At 100K GPUs, checkpoint and restart time decide your goodput. A worked model of checkpoint size, bandwidth, and optimal interval, plus what it costs when you get it wrong.
Big training runs don't fail once in a while. They fail constantly, and the cluster's job is to keep going anyway. Checkpointing is the mechanism, and storage is usually where it breaks down. "We have plenty of NFS" isn't a strategy. Here's how to size it with real numbers.
TL;DR
- Meta's Llama 3 405B run on up to 16,384 H100s logged 466 interruptions in 54 days, 419 of them unexpected: about one every ~3 hours (Llama 3 paper).
- Failure rate scales with GPU count. At ~65K GPUs, the same per-GPU reliability means an interruption every ~47 minutes.
- Using the classic Young/Daly model, going from a 30 s blocking save to ~5 s async staging recovers several points of goodput at 16K GPUs and much more at 64K. Restart time ends up mattering even more.
How often things break
The Llama 3 paper is still the best public dataset on this. Over a 54-day snapshot: 47 planned interruptions and 419 unexpected ones. 58.7% of the unexpected ones were GPU-related, including 148 GPU faults and 72 HBM3 faults. Meta still kept over 90% effective training time through automated recovery and faster checkpointing (Meta, 2024).
The key point is that MTBF for a synchronous job scales roughly as 1/N. Any one GPU, HBM stack, NIC, optic, or host failing stops every rank. Double the GPUs and you roughly halve the time between failures.
How big is a checkpoint?
For dense mixed-precision training with Adam, a full checkpoint usually holds per parameter:
- BF16 weights: 2 bytes
- FP32 master weights: 4 bytes
- Adam first and second moments: 4 + 4 bytes
That's ~12–14 bytes per parameter, depending on what your framework saves.
| Model | Params | Checkpoint (~14 B/param) |
|---|---|---|
| Llama-class 70B | 70B | ~1 TB |
| Llama-class 405B | 405B | ~5.7 TB |
| 1T-param MoE (all experts) | 1T | ~14 TB |
To write 5.7 TB in 60 seconds you need ~95 GB/s of sustained aggregate write. Spread over 16K GPUs that's only ~6 MB/s per GPU, which is trivial. The hard part isn't per-node bandwidth. It's every rank hitting the storage system at the same moment, then all of them reading at once on restart.
The math: Young/Daly
The standard result for the optimal checkpoint interval is τ ≈ √(2·C·M), where C is the time training is blocked per checkpoint and M is MTBF (Daly, 2006). Expected waste ≈ C/τ (time spent saving) + (τ/2 + R)/M (lost work plus restart time R per failure).
I plugged in Llama 3's failure rate scaled by GPU count and a 10-minute restart (rescheduling, NCCL init, reloading state, which is optimistic for many teams):
| GPUs | MTBF | Waste, 120 s blocking | Waste, 30 s blocking | Waste, 5 s async |
|---|---|---|---|---|
| 4,096 | ~12.4 h | 8.7% | 5.0% | 2.8% |
| 16,384 | ~3.1 h | 20.0% | 12.7% | 8.4% |
| 65,536 | ~47 min | 50.8% | 36.2% | 27.5% |
Two takeaways:
- Async checkpointing is table stakes. PyTorch DCP's async save copies state GPU→host, then persists in background threads. IBM measured blocked time for a 7B model dropping from 148.8 s to 6.3 s (PyTorch blog).
- At large scale, restart time R dominates. At 64K GPUs, even a 5 s save leaves ~27% waste, mostly from restarts. Fast detection, hot spares, in-memory or peer checkpoints (restoring from a neighbor's host RAM rather than storage), and fast NCCL re-init are what move the number from there.
The money
Put a price on it. At SemiAnalysis's March 2026 H100 one-year rate of ~$2.35/GPU-hr (SemiAnalysis), a 16,384-GPU cluster costs about $38.5K per hour, or ~$640 per minute.
- Checkpoint every 15 minutes with a 60 s blocking save → 96 stalls/day → ~$62K/day, ~$22M/year of GPUs sitting idle while they write.
- Cut that to ~5 s and you get almost all of it back.
Fast storage that saves even a few points of goodput pays for itself very quickly on a cluster like this.
A practical architecture
- Tier 0: host DRAM. Async-stage shards from HBM to pinned host memory. Size host RAM with headroom for at least one full local shard.
- Tier 1: local NVMe. Persist from DRAM to node-local flash. NVIDIA's DGX Vera Rubin compute trays ship with four E1.S NVMe cache drives each for this kind of job (DGX VR NVL72 guide).
- Tier 2: parallel FS / object. Drain asynchronously to durable shared storage. This tier needs aggregate bandwidth for the drain, plus a restore burst when every rank reads at once.
- Retention. Keep the last few checkpoints hot and older ones in cheaper object storage. Even 10 copies of a 5.7 TB checkpoint is only ~57 TB.
Where teams get cooked
- Sizing for write and forgetting read. Restore is the worst burst: all ranks, at once, right after a failure.
- Checkpointing on the training process group. That can deadlock collectives. Use a separate group (the PyTorch team calls this out).
- Not checkpointing data-loader and RNG state. You resume, but on different data. Silent L.
- Never testing restore. A checkpoint you haven't restored is just a hope. Run restore drills on a schedule.
Bottom line
Checkpoint storage isn't a capacity problem. It's a burst-bandwidth and restart-latency problem, and it gets worse the more GPUs you add. Do the Young/Daly math for your own MTBF, make saves async, and spend your engineering effort on restart time. That's where the goodput is.