Same GPUs, very different clusters. Training is a synchronized throughput problem. Inference is a memory-bandwidth and KV-cache problem, which is why prefill/decode disaggregation is taking over. The math, with Llama 3 70B as the example.
#training#inference#gpu-clusters#kv-cache#economics