InfiniBand: from SDR to XDR

The business question

You are about to approve a cluster worth several hundred thousand dollars and, somewhere in the spreadsheet, a line called “interconnect” appears costing between 15% and 25% of the total. Someone will say there is room to save there.

That is one of the few budget lines where saving destroys the entire asset. This text explains why, without requiring you to be a network engineer.

What InfiniBand actually does

The problem it solves

In a conventional network, when one server sends data to another, the data travels from disk or memory up to the operating system, is copied into a kernel work area, packetised, handed to the network card, and the reverse path happens on the other side. Each of those steps costs time and, worse, costs processor, the very processor that should be coordinating your calculation.

When the work is a simulation or a distributed training run, that cycle repeats millions of times. What was a detail becomes the bottleneck.

InfiniBand was designed, from the outset, to eliminate that path. The network card, called an HCA (Host Channel Adapter), is granted permission to read and write directly into the remote server’s memory. The operating system authorises this once, at the beginning, and then steps aside. It is called RDMA (Remote Direct Memory Access).

Two practical consequences:

• Latency collapses, because the path is physically shorter.
• The processor is freed, because the transfer happens without it. On a machine with eight GPUs, that means the CPU keeps feeding the GPUs instead of managing packets.

Why this matters more in AI than in any other workload

Training a large model across many GPUs requires that, at every step, all GPUs exchange and combine their partial results. It is a collective operation: nobody moves forward until the last one finishes.

The effect is relentless. If the GPUs spend 30% of their time waiting on the network, you bought a cluster and are using 70% of it, permanently, for every year of the asset’s life. It is the same as paying for eight GPUs and receiving five and a half.

InfiniBand attacks this on two fronts: it reduces the time of each exchange and, in recent generations, performs part of the combination inside the switch itself, a technology NVIDIA calls SHARP. Instead of all the data travelling to one node, being summed and returning, the sum happens along the way.

The three structural guarantees

Three characteristics, present from conception, separate InfiniBand from an adapted office network:

Lossless fabric. In classic Ethernet, when congestion hits, the packet is dropped and retransmitted. InfiniBand uses credit-based flow control: the sender only transmits once the receiver has guaranteed room. The packet is not dropped because it is never sent without a guaranteed destination. In a collective operation, a single retransmission delays the entire group.

Centralised management. A component called the Subnet Manager sees the whole topology, computes the routes and distributes them. The network is planned, not discovered, which makes behaviour predictable under load.

Adaptive routing. When a path congests, traffic is diverted in real time, without waiting for the application layer to notice.

Reference table · The InfiniBand generations

Each generation doubles the rate per lane. The most common link width in a cluster is x4, four lanes grouped into one port. That is why “NDR” means 400 Gb/s in practice, even though the individual lane runs at 100 Gb/s.

Two readings the table hides, and both are worth money:

Encoding is not a detail. Up to QDR, for every 10 bits transmitted, 2 were synchronisation overhead, 20% of the link. That is why QDR, advertised as “40 Gb/s”, delivers 32. From FDR onward, 64b/66b encoding drops the overhead to roughly 3%. If you compare old equipment with new by the number on the datasheet, you are comparing different quantities.

PAM4 changes the physics. Up to EDR, the signal was binary: high or low. From HDR onward, each symbol carries four levels, two bits per symbol. The rate doubles without doubling the frequency, but the levels sit closer together and noise tolerance falls. Direct consequence: mandatory error correction (which adds latency) and shorter copper reach. Physics is the reason your new cluster needs more fibre than the old one.

Typical end-to-end latency at NDR: [DATA TBC], to be filled with our own measurement or cited NVIDIA documentation; do not publish a manufacturer figure without attribution.

The ecosystem that runs on InfiniBand

InfiniBand rarely appears alone in the conversation. These are the names you will hear and what each one means:

“NVMe over InfiniBand” is the informal name for the NVMe-oF over RDMA over InfiniBand combination.

The interfaces and the cables

Connectors

The connector changes with the generation because the physics changes. The rule of thumb: QSFP was the dominant family from QDR through HDR; OSFP, larger and with far greater thermal dissipation, became necessary at NDR; OSFP-XD is the answer for 1.6 Tb/s.

One detail causes real confusion: at NDR, a switch OSFP cage frequently carries two 400 Gb/s ports. A switch with 32 OSFP cages offers 64 ports of 400 Gb/s. Counting cages instead of ports undersizes the switch by half.

The three media

The trend that decides the project: with every generation, the useful reach of copper shrinks. What was solved with cheap copper at EDR requires active copper at HDR and optics at NDR. Interconnect cost per port rises faster than the transmission rate, and that has to enter the budget before the purchase, not after.

In the optical variants, the suffix states the reach: SR short on multimode fibre, DR around half a kilometre, FR a few kilometres, LR tens.

The math that decides

Consider a cluster of 32 nodes with 8 GPUs each, 256 GPUs, with a GPU acquisition cost in the region of US$ 8M, an illustrative figure, replace it with your own.

Scenario A, saving on the network. The interconnect is sized below what is needed. The GPUs sit idle 25% of the time waiting on communication. Savings at purchase: say US$ 400K.

Scenario B, properly sized network. Idleness falls to 8%.

The calculation that matters:

• Difference in utilisation: 17 percentage points on an US$ 8M asset.
• GPU value effectively lost in Scenario A: US$ 1.36M of capacity you bought and do not use.
• Across a 4-year service life, the undersized cluster delivers the work of a significantly smaller cluster, while consuming the same power, occupying the same space and costing the same support contract.

The US$ 400K saved cost US$ 1.36M of installed capacity. And unlike almost every other specification error, this one is irreversible without replacing the cabling and the adapters, which is to say, without stopping the cluster.

There is a second effect, less visible and equally expensive: a cluster with poor communication does not scale. Adding nodes improves little, because the coordination cost grows faster than the capacity. You lose not only performance today, but the option to grow tomorrow.

This is the reasoning behind the architecture principle ARETE applies: specify the interconnect for the frontier, not for today’s workload. The network is the longest-lived subsystem in the cluster and the most expensive to replace. The compute nodes will be replaced two or three times; the structured cabling, almost never.

Common specification mistakes

1 · Confusing nominal rate with effective rate. “QDR 40 Gb/s” delivers 32. It is 8b/10b encoding, not misleading marketing, but whoever compares generations by the datasheet compares different quantities.

2 · Reading “1.6 Tb/s” as a 1.6 Tb/s port. In the overwhelming majority of cases, the number refers to a dual-port module (2 × 800 Gb/s), to an aggregate capacity, or to the future roadmap (GDR). Always ask: is this per port, per cage, or aggregate? The answer changes the sizing of the switch.

3 · Counting OSFP cages as if they were ports. At NDR, one cage commonly serves two 400 Gb/s ports. Getting this wrong undersizes the switch by 50%.

4 · Planning copper where the generation already demands optics. Copper reach shrinks with every step. A room layout inherited from the previous generation, reused without review, turns into an emergency purchase of transceivers, at the worst moment, at the worst price.

5 · Treating the network as a component rather than a chain. A fast adapter behind an undersized switch delivers the performance of the switch. An excellent switch behind inadequate cabling delivers the performance of the cable. The network is a chain in series: the result is that of the weakest link, and the weakest link is rarely the component that appears in the proposal.