RDMA, RoCEv2 and InfiniBand
Kernel bypass, queue pairs and GPUDirect RDMA; why a lossless Ethernet fabric needs PFC, ECN and DCQCN; and how RoCEv2 and InfiniBand genuinely differ.
Conventional networking spends CPU cycles on every packet: copying data between application memory and kernel buffers, walking a protocol stack, handling interrupts. At the volumes an AI cluster generates, that overhead is not an inefficiency to tolerate — it is a wall. RDMA is how the industry gets around it, and this chapter covers what that costs in return.
Getting the terms right
- RDMA
- Remote Direct Memory Access.
- RoCE
- RDMA over Converged Ethernet.
- RoCEv2
- Routable RDMA, encapsulated in UDP/IP.
- InfiniBand
- An established high-performance interconnect architecture.
How RDMA works
The network adapter reads and writes application memory directly. The operating system sets the transfer up, then gets out of the way — it is not involved in moving each piece of data.
Architecture
- Kernel bypass
- The application communicates with the adapter directly rather than through kernel system calls on the data path.
- Reduced-copy data movement
- Data moves from application memory to the wire without the intermediate copies a conventional stack performs.
- Queue pairs
- A send queue and a receive queue forming the connection abstraction. The application posts work requests to them.
- Completion queues
- Where the adapter reports finished work, so the application can poll for completion rather than wait on interrupts.
- GPUDirect RDMA
- Transfers that land directly in accelerator memory, removing the host-memory staging step entirely.
Engineering deep dive
RoCEv2 and lossless Ethernet
RoCEv2 carries RDMA inside ordinary UDP/IP packets, so it routes across a normal IP fabric. The difficulty is that ordinary Ethernet drops packets when congested, and RDMA would rather it did not.
Architecture
The three mechanisms work at different timescales. ECN asks senders to slow down. DCQCN turns those signals into a rate-control loop at the endpoint. PFC is the last resort that stops a link losing packets outright — and the one with the worst side effects.
Engineering deep dive
| Mechanism | What it does | What it costs |
|---|---|---|
| ECN | Marks packets to signal congestion instead of dropping them. | Requires the endpoint to react; the signal takes a round trip to have effect. |
| DCQCN | Converts ECN marks into endpoint rate control. | Needs tuning; poorly tuned parameters either react too slowly or oscillate. |
| PFC | Pauses a priority class on a link to prevent loss. | Pause propagates backwards, causing head-of-line blocking and spreading congestion far from its source. |
Conceptual. Actual thresholds and behaviour depend on platform, buffer architecture and configuration.
- Buffers — where congestion is absorbed. Deeper buffers hide bursts but add latency and delay the congestion signal.
- Head-of-line blocking — one paused or slow flow holding up unrelated traffic behind it.
- Rail-optimised networking — placing each accelerator on a specific fabric plane so peers communicate within a rail, reducing cross-fabric traffic.
- Multi-plane networking — parallel fabric planes providing both capacity and failure isolation.
- Failure handling — a rail loss should degrade bandwidth, not fail the job, provided the design and the job both accommodate it.
Operational implications
What operations teams actually watch: PFC pause counters, ECN marking rates, buffer occupancy, and per-node throughput against a commissioning baseline. Pause counters rising across many ports simultaneously is the signature of a storm — and the reporting ports are symptoms, not the cause.
InfiniBand
InfiniBand provides RDMA natively rather than layering it over a network designed for something else. It has its own link layer, its own congestion mechanisms, and a subnet manager that owns topology and routing centrally rather than distributing that responsibility across an IP control plane. It has been the default in HPC for a long time; AI clusters adopted it rather than inventing it.
Comparing the three
| Dimension | TCP/IP Ethernet | RoCEv2 Ethernet | InfiniBand |
|---|---|---|---|
| Typical role | General networking | Ethernet AI fabric | HPC and AI fabric |
| RDMA | Not normally | Yes | Native |
| Routing | IP | IP-routable | Fabric-specific, with a subnet manager |
| Operations | Widely understood | Requires careful tuning | Specialist fabric skills |
| Congestion | Standard Ethernet mechanisms | AI-focused loss and congestion controls (ECN, DCQCN, PFC) | Native fabric mechanisms |
| Ecosystem | Very broad | Broad and expanding | Mature HPC/AI ecosystem |
A conceptual comparison, not a universal verdict. The right choice depends on scale, operating model and the skills the team actually has.
Key Takeaways
- RDMA lets the adapter read and write application memory directly, removing per-packet CPU cost from the data path.
- RDMA protocols assume a network that does not drop packets, which is why lossless Ethernet mechanisms exist at all.
- RoCEv2 is routable because it encapsulates RDMA in UDP/IP — that is what distinguishes it from RoCEv1.
- ECN signals congestion by marking; DCQCN turns that into endpoint rate control; PFC pauses links as a last resort.
- A well-tuned fabric makes ECN act before PFC has to, because pause propagates backwards and spreads congestion.
- In a PFC storm, the ports reporting pause are symptoms — the congestion source is elsewhere.
- Disabling PFC to stop a storm converts a slowdown into packet loss and makes it worse.
- InfiniBand is long-established, not emerging; the RoCEv2-versus-InfiniBand choice is largely about operating model and skills.
- "RoCI" is not a technology. The term is RoCE.