Tom’s Tech Solutions
← All writing

HGX to NVL72: What Changes When the Rack Becomes the Server

An 8-GPU HGX node and a 72-GPU NVL72 rack look like the same idea at different sizes. They aren't. Here's what actually changes in the NVLink domain, the network, power, cooling, and how you plan a deployment.

Tom Pechulis5 min read

For most of the last few years, the unit of AI compute was the server. You bought HGX nodes, eight GPUs each, wired them into a fabric, and the cluster was "N nodes times eight." With GB200 and GB300 NVL72, the unit is the rack. That sounds like a packaging change. In practice it changes how you design the network, how you plan power and cooling, how you think about failure, and what a "small" deployment even is.

This post walks through the differences that matter when you're planning a build, not the marketing slides.

The HGX baseline

An HGX board puts eight GPUs on a single baseboard, connected through NVSwitch so every GPU can talk to every other GPU over NVLink at full bandwidth. On H100, that's NVLink 4 at 900 GB/s of bidirectional bandwidth per GPU. On B200, NVLink 5 doubles it to 1.8 TB/s per GPU.

The important number is eight. That's the size of the NVLink domain: the set of GPUs that can share memory and run tensor-parallel work at NVLink speed. Anything beyond those eight GPUs goes out through the NICs and over the scale-out network, typically InfiniBand or RoCE Ethernet at 400 Gb/s per GPU, which is roughly an order of magnitude less bandwidth than NVLink.

That gap shapes how models get split. Tensor parallelism, which is chatty and latency-sensitive, stays inside the node. Pipeline and data parallelism, which tolerate the slower link, stretch across nodes.

What NVL72 actually is

A GB200 NVL72 rack holds:

  • 18 compute trays, each with two GB200 superchips. Each superchip pairs one Grace CPU with two Blackwell GPUs, so a tray is 4 GPUs and 2 CPUs.
  • 9 NVLink switch trays that connect all 72 GPUs into one NVLink domain.
  • A copper NVLink spine in the back of the rack tying the trays to the switches.

Total: 72 Blackwell GPUs and 36 Grace CPUs acting as a single NVLink domain, with about 130 TB/s of aggregate NVLink bandwidth. GB300 NVL72 keeps the same rack-scale design and moves to Blackwell Ultra GPUs with more HBM per GPU (288 GB) and ConnectX-8 SuperNICs at 800 Gb/s.

So the NVLink domain goes from 8 GPUs to 72. That's the whole point of the product.

With HGX, the scale-up domain is a box you can rack anywhere. With NVL72, the scale-up domain is a rack you have to land as a complete system. A few consequences:

  • Model parallelism can stretch further on NVLink. Large mixture-of-experts models, where expert-parallel traffic is all-to-all and heavy, benefit the most. Inference for big models gets a lot of headroom from having 72 GPUs' worth of HBM in one NVLink domain.
  • You don't buy "half a rack." Your capacity planning moves in steps of 72 GPUs. Plenty of teams that would have started with four HGX nodes (32 GPUs) now face a choice between one rack or a different platform.
  • The rack becomes the failure and maintenance domain. A failed compute tray doesn't take down the rack, but the scheduler and the training framework need to understand that the NVLink domain is degraded. Plan for spare trays and for jobs that can restart on a smaller domain.

Change 2: the scale-out network shrinks in relative importance, not in size

It's tempting to assume that a bigger NVLink domain means a smaller back-end network. It doesn't. Each GPU still gets its own high-speed NIC into the scale-out fabric, so a 1,000-rack deployment still has tens of thousands of 400G or 800G ports to cable, switch, and optic.

What changes is the traffic pattern. Collectives that used to cross the scale-out fabric between two 8-GPU nodes now stay inside the rack. The scale-out fabric carries mostly data-parallel and pipeline traffic. That makes the fabric's job more predictable, but it doesn't make it optional, and rail-optimized topologies, congestion control, and optics planning still dominate the network bill of materials.

Change 3: power density jumps

An 8-GPU HGX H100 server is rated around 10 kW. Four of those in an air-cooled rack lands around 40 kW, which already pushed a lot of older facilities past their limits.

An NVL72 rack is on the order of 120 to 140 kW. That's not an incremental step. Most existing colocation halls were designed for 10 to 20 kW per rack, so NVL72 usually means either a purpose-built hall or a significant retrofit: new busway, new PDUs, and a lot more power delivered to a much smaller footprint.

Change 4: liquid cooling stops being optional

You can't air-cool 130 kW in a rack. NVL72 uses direct-to-chip liquid cooling: cold plates on the GPUs and CPUs, manifolds in the rack, and a coolant distribution unit (CDU) that moves heat from the rack's liquid loop to the facility water loop.

What that means for planning:

  • The facility needs a water loop sized for it, at the right supply temperature and flow rate, with redundancy you trust.
  • Some heat still goes to air. Power supplies, NICs, and other components are often still air-cooled, so the room still needs air handling, just much less of it per kW.
  • Commissioning gets harder. Leak detection, pressure testing, coolant quality, and filtration become part of the bring-up checklist, alongside the usual burn-in and NCCL tests.

Change 5: delivery and bring-up look different

HGX nodes ship as servers. NVL72 typically ships as an integrated, pre-tested rack. That's better in some ways: the NVLink spine is already cabled, and the rack has been validated at the factory. It's harder in others: a fully loaded rack weighs well over a metric ton, so floor loading, loading docks, freight elevators, and the path from the dock to the row all have to be checked in advance.

When HGX is still the right call

NVL72 isn't automatically better. HGX still makes sense when:

  • Your facility is air-cooled and can't support 100+ kW racks.
  • Your workloads fit comfortably in an 8-GPU domain, such as fine-tuning, smaller-model inference, or many independent jobs.
  • You want to grow in smaller increments, or you need to place compute across many sites.

The takeaway

Going from HGX to NVL72 isn't about getting more GPUs per rack. It's about moving the scale-up boundary from the server to the rack. Everything else follows from that: you plan capacity in 72-GPU steps, you design facilities around liquid and around 120+ kW racks, and you build schedulers and runbooks that treat the rack as the system.

If you're scoping a deployment, start with the facility, not the GPU count. The power and cooling you can actually deliver decide which platform you can run.