Unlocking HPC: Why InfiniBand is the Preferred Choice for RDMA

Posted on May 22, 2024 by

When it comes to high-performance computing (HPC), speed and efficiency are paramount. Remote Direct Memory Access (RDMA) is a technology that enables memory in one computer to be accessed by another computer without involving the operating system or CPU, which drastically reduces latency and improves throughput. While there are several RDMA technologies, InfiniBand has established itself as the leader in this space. This article highlights the reasons why InfiniBand stands out as the technology of choice for RDMA applications.

SDN Principles: Enhanced Network Efficiency and Simplicity

One of InfiniBand's standout features is its alignment with the principles of Software-Defined Networking (SDN). Managed by a subnet manager acting as the SDN controller, InfiniBand eliminates the need for routing protocols traditionally found in Ethernet networks, including RoCE. The entire network's forwarding tables are computed and distributed by a centralized subnet manager. Additionally, the subnet manager handles crucial configuration aspects within the InfiniBand subnet, such as partitioning and Quality of Service (QoS). Unlike RoCE, InfiniBand networks do not rely on broadcast mechanisms like ARP for forwarding table learning, effectively eliminating broadcast storms and unnecessary bandwidth consumption.



In contrast, traditional Ethernet, including RoCE, does support SDN controller-based networking. However, network vendors have shifted away from the earlier OpenFlow-based flow table forwarding concept to avoid becoming mere "white-box" manufacturers. Instead, they have embraced solutions based on netconf, VXLAN, and EVPN. While SDN controllers have evolved into advanced "network management systems" focused on deploying control policies, forwarding still heavily relies on device-based learning, such as MAC table learning, ARP tables, and routing tables. This divergence has resulted in RoCE losing the efficiency and simplicity advantages found in InfiniBand.



Congestion Avoidance Mechanism: Achieved Native Lossless Network

InfiniBand networks leverage a credit-based mechanism that effectively prevents buffer overflow and packet loss issues. This mechanism ensures that packet transmission is initiated only when the receiver has sufficient credits to accept the corresponding number of messages.

The credit-based mechanism functions as follows: Each link in the InfiniBand network has a predetermined buffer for storing packets to be transmitted. Before sending data, the sender checks the receiver's available credits, representing the current buffer size. Based on this credit value, the sender determines whether to initiate packet transmission. If the receiver has insufficient credits, the sender waits until the receiver releases enough buffer space and reports new available credits.

Once the receiver finishes forwarding, it releases the utilized buffer and continuously reports the current available buffer size to the sender. This dynamic adjustment enables the sender to fine-tune packet transmission based on the receiver's buffer status. Such link-level flow control ensures that the sender does not overwhelm the receiver with excessive data, effectively preventing network buffer overflow and packet loss.

In contrast, RoCE employs a "post-congestion" management mechanism. It does not negotiate resources with the receiver before sending packets but directly forwards them without prior coordination. Only when the receiver's switch experiences port buffer congestion (or imminent congestion) does it send congestion management messages using Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) protocols to reduce or pause packet transmission on the opposing switch and network card. While this "post-congestion" approach can partially alleviate congestion impact, it falls short of completely preventing packet loss and maintaining network stability.

Schematic diagram of lossless data transmission InfiniBand network

Direct Forwarding Mode: Enabled Networks to Achieve Lower Latency

Ethernet networks, like those utilizing RoCE, generally use a store-and-forward method where switches buffer the incoming data packet, check its destination and integrity, and then send it on. This can lead to delays, especially under heavy packet traffic loads.

Conversely, switches with Cut-through technology scan only the packet's header to identify the port for delivery, beginning the forwarding process at once. This slashes the packet's dwell time in the switch and minimizes latency.

InfiniBand switches employ such Cut-through forwarding, streamlining the process for layer 2 packets. They quickly pin down the routing path using a 16-bit LID from the subnet manager, dropping latency below 100 nanoseconds. Ethernet switches, conversely, rely on MAC table lookups and store-and-forward techniques that take longer owing to additional tasks like handling IP and MPLS. While some Ethernet switches might use Cut-through, their delays can still be over 200 nanoseconds.

Cut-through forwarding

Professional Provider of Networking Solutions-FS

FS offers comprehensive solutions with its selection of InfiniBand optical modules and high-speed cables. Below are the original IB products available from FS.
PCIe Interface
32 Ports / 40 Ports
40 x HDR 200G QSFP56 /
32 x NDR 800G OSFP
Managed / Unmanaged
Dual / Single
100G QSFP56
200G QSFP56
400G QSFP112
PCIe 4.0 x 16
PCIe 5.0 x 16
ConnectX®-6 VPI
ConnectX®-7 VPI
800G NDR / 400G NDR / 200GHDR / 100G EDR / 56G FDR / 40G FDR
≦50m distance ≦100m
800G NDR / 400G NDR / 200GHDR / 100G EDR / 56G FDR / 40G FDR
≦50m distance ≦40km

Quick Shipping: Our ample inventory guarantees speedy delivery. Orders are reliably dispatched ASAP, accelerating your project’s timeline by minimizing delays.

Test Support: FS provides professional performance test, reliability test, scenario test,and compatibility test. If you want to find out more about testing, please go to Test Support Center.

Reliable Support and Delivery: We have a track record of stable product functioning that has earned customer trust through multiple successful partnerships. Our agile technical support offers peace of mind with continual after-sales assistance.

Rest assured, our consistent stock levels cater to your immediate needs, eliminating concerns about quality or availability.


InfiniBand's native lossless networking and superior flow control make it the preferred choice for RDMA deployments. By leveraging these unique advantages, businesses can optimize their networking infrastructure and enhance overall performance.

You might be interested in

See profile for Howard.
InfiniBand Network and Architecture Overview
Dec 30, 2023
See profile for Virginia.
What Is High-Performance Computing (HPC)?
Jun 16, 2022
See profile for Sheldon.
Decoding OLT, ONU, ONT, and ODN in PON Network
Mar 14, 2023
See profile for Irving.
What's the Difference? Hub vs Switch vs Router
Dec 17, 2021
See profile for Sheldon.
What Is SFP Port of Gigabit Switch?
Jan 6, 2023
See profile for Migelle.
PoE vs PoE+ vs PoE++ Switch: How to Choose?
May 30, 2024