#training
- 4 min read
Training vs Inference: Two Different Infra Problems
Same GPUs, very different clusters. Training is a synchronized throughput problem. Inference is a memory-bandwidth and KV-cache problem, which is why prefill/decode disaggregation is taking over. The math, with Llama 3 70B as the example.
#training#inference#gpu-clusters#kv-cache#economics
- 4 min read
Checkpoint Storage That Doesn't Stall Training
Meta's Llama 3 run hit an interruption every ~3 hours on 16K GPUs. At 100K GPUs, checkpoint and restart time decide your goodput. A worked model of checkpoint size, bandwidth, and optimal interval, plus what it costs when you get it wrong.
#storage#training#gpu-clusters#reliability#economics