Unlocking HPC: Why InfiniBand is the Preferred Choice for RDMA

When it comes to high-performance computing (HPC), speed and efficiency are paramount. Remote Direct Memory Access (RDMA) is a technology that enables memory in one computer to be accessed by another computer without involving the operating system or CPU, which drastically reduces latency and improves throughput. While there are several RDMA technologies, InfiniBand has established itself as the leader in this space. This article highlights the reasons why InfiniBand stands out as the technology of choice for RDMA applications.

One of InfiniBand's standout features is its alignment with the principles of Software-Defined Networking (SDN). Managed by a subnet manager acting as the SDN controller, InfiniBand eliminates the need for routing protocols traditionally found in Ethernet networks, including RoCE. The entire network's forwarding tables are computed and distributed by a centralized subnet manager. Additionally, the subnet manager handles crucial configuration aspects within the InfiniBand subnet, such as partitioning and Quality of Service (QoS). Unlike RoCE, InfiniBand networks do not rely on broadcast mechanisms like ARP for forwarding table learning, effectively eliminating broadcast storms and unnecessary bandwidth consumption.

In contrast, traditional Ethernet, including RoCE, does support SDN controller-based networking. However, network vendors have shifted away from the earlier OpenFlow-based flow table forwarding concept to avoid becoming mere "white-box" manufacturers. Instead, they have embraced solutions based on netconf, VXLAN, and EVPN. While SDN controllers have evolved into advanced "network management systems" focused on deploying control policies, forwarding still heavily relies on device-based learning, such as MAC table learning, ARP tables, and routing tables. This divergence has resulted in RoCE losing the efficiency and simplicity advantages found in InfiniBand.

InfiniBand networks leverage a credit-based mechanism that effectively prevents buffer overflow and packet loss issues. This mechanism ensures that packet transmission is initiated only when the receiver has sufficient credits to accept the corresponding number of messages.

The credit-based mechanism functions as follows: Each link in the InfiniBand network has a predetermined buffer for storing packets to be transmitted. Before sending data, the sender checks the receiver's available credits, representing the current buffer size. Based on this credit value, the sender determines whether to initiate packet transmission. If the receiver has insufficient credits, the sender waits until the receiver releases enough buffer space and reports new available credits.

Once the receiver finishes forwarding, it releases the utilized buffer and continuously reports the current available buffer size to the sender. This dynamic adjustment enables the sender to fine-tune packet transmission based on the receiver's buffer status. Such link-level flow control ensures that the sender does not overwhelm the receiver with excessive data, effectively preventing network buffer overflow and packet loss.

In contrast, RoCE employs a "post-congestion" management mechanism. It does not negotiate resources with the receiver before sending packets but directly forwards them without prior coordination. Only when the receiver's switch experiences port buffer congestion (or imminent congestion) does it send congestion management messages using Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) protocols to reduce or pause packet transmission on the opposing switch and network card. While this "post-congestion" approach can partially alleviate congestion impact, it falls short of completely preventing packet loss and maintaining network stability.

Ethernet networks, like those utilizing RoCE, generally use a store-and-forward method where switches buffer the incoming data packet, check its destination and integrity, and then send it on. This can lead to delays, especially under heavy packet traffic loads.

Conversely, switches with Cut-through technology scan only the packet's header to identify the port for delivery, beginning the forwarding process at once. This slashes the packet's dwell time in the switch and minimizes latency.

InfiniBand switches employ such Cut-through forwarding, streamlining the process for layer 2 packets. They quickly pin down the routing path using a 16-bit LID from the subnet manager, dropping latency below 100 nanoseconds. Ethernet switches, conversely, rely on MAC table lookups and store-and-forward techniques that take longer owing to additional tasks like handling IP and MPLS. While some Ethernet switches might use Cut-through, their delays can still be over 200 nanoseconds.