Scale-up versus scale-out networking
Two different communication problems that are constantly confused: tightly-coupled traffic inside a compute domain, and traffic between racks and pods.
These two terms are used constantly and confused almost as often. They describe genuinely different communication problems with different physics, different technologies and different design consequences. Getting them straight is the single most useful piece of vocabulary work in AI networking.
Side by side
| Scale-up | Scale-out | |
|---|---|---|
| The question it answers | How large can one tightly coupled unit be? | How many such units can cooperate? |
| Scope | Within a tray, rack or coupled domain | Between nodes, racks and pods |
| Technologies | PCIe, NVLink, NVLink switches | Ethernet with RoCEv2, InfiniBand |
| Traffic | Accelerator-to-accelerator, collective operations, coordinated memory access | East-west between nodes, plus storage access |
| Routing | Not routed — a coupled domain | Routed, with equal-cost multipath across a Clos fabric |
| Failure domain | Losing the domain fragments it into individual trays | Losing a link or switch reduces capacity, usually without an outage |
| Bounded by | Physics, packaging, power and cooling | Port count, optics, cost and operational tolerance |
| Design consequence | Sets the largest model partition that fits in tightly coupled memory | Sets how gracefully large jobs degrade under contention |
A conceptual orientation. Specific technologies vary by platform and reference architecture.
Scale-up in more detail
Inside a tightly coupled domain, accelerators exchange data at very high bandwidth and very low latency. Practically, they behave more like parts of one machine than like separate computers on a network.
Architecture
- PCIe — the host bus connecting processors and adapters within a system. Present in every design, and a real bottleneck in some.
- NVLink — high-bandwidth accelerator-to-accelerator interconnect within the domain.
- NVLink switches — extend the domain beyond a single tray, which is what allows a whole rack to behave as one coupled unit.
- Collective communication — all-reduce and similar operations that keep participants in step.
- Coordinated memory access — patterns that would be impractical over a routed network.
Engineering deep dive
Operational implications
A degraded scale-up link does not usually cause a clean failure. It shrinks the usable domain or slows collectives, so large jobs stop being schedulable or simply run slower. Validating reported topology against intended topology after every physical intervention is the control that catches this.
Scale-out in more detail
Between coupled domains, communication crosses a routed network — the same basic idea as any data-centre fabric, but carrying far more east-west traffic and far less tolerance for loss.
Architecture
- Ethernet with RoCEv2 or InfiniBand as the transport.
- Spine-leaf (Clos) fabrics giving predictable path counts between any two endpoints.
- Multi-rail designs, where each accelerator connects to a specific fabric plane so peers communicate within the same rail.
- GPUDirect RDMA, so transfers land directly in accelerator memory rather than staging through the host.
Engineering deep dive
The defining difficulty is that AI traffic is synchronised. Thousands of endpoints transmit simultaneously at the same point in every training step. A fabric sized for average utilisation behaves quite differently from one sized for that burst, and congestion-control behaviour matters far more than in a conventional data centre. The RDMA and spine-leaf fabric sections cover the mechanics and the arithmetic.
Operational implications
Compare the two domains interactively
These cards describe structure and roles. The figures — 72 Rubin GPUs and 36 Vera CPUs per rack, memory capacities, NVLink and scale-out bandwidths, weight and availability — are in the Vera Rubin NVL72 specification behind the ⓘ on The Compute Rack, each sourced to NVIDIA's own pages; they are vendor figures, not independently verified. NVIDIA has not published a rack power figure for Vera Rubin NVL72, so none is given. Announced functionality is not the same as generally available functionality.
The platform as a coordinated whole: host processing, acceleration, the two networking domains, infrastructure offload, storage and the software that binds them.
Key Takeaways
- Scale-up asks how large one tightly coupled unit can be; scale-out asks how many such units can cooperate.
- NVLink and PCIe serve scale-up; Ethernet with RoCEv2 and InfiniBand serve scale-out.
- The scale-up domain boundary determines whether a model partition runs at interconnect bandwidth or crosses a routed fabric.
- AI scale-out traffic is synchronised, so a fabric sized for average utilisation behaves very differently under a real collective.
- Scale-up faults shrink or fragment the domain; scale-out faults usually degrade throughput without failing anything.
- Adding scale-out bandwidth cannot substitute for a larger scale-up domain, and vice versa.