InfiniBand Insights: Powering High-Performance Computing in the Digital Age

Since the early 21st century, data centers have rapidly evolved, driven by the rise of cloud computing and big data. While Ethernet remains the primary technology used in data centers, InfiniBand technology is increasingly being adopted for high-performance computing(HPC) environments and data centers. This article will focus on discussing InfiniBand technology, exploring its key features and its role in high-performance computing.

The Evolutionary Journey of InfiniBand Technology

The birth of InfiniBand is closely associated with the evolution of computer architecture.

In the early 1990s, to meet the rapidly increasing demand for external device support in the computing field, Intel introduced the Peripheral Component Interconnect (PCI) bus design into the standard PC architecture, laying the foundation for the emergence of InfiniBand.

However, as online businesses and user bases grew exponentially, PCI bus upgrades lagged compared to components like CPUs, memory, and hard drives. This severely limited I/O performance and became the bottleneck of the entire system.

To solve this issue, Intel, Microsoft, and Sun Microsystems led the development of the "Next Generation I/O (NGIO)" technical standard. Meanwhile, IBM, Compaq, and HP led the development of "Future I/O (FIO)".

In 1999, the FIO Developers Forum and NGIO Forum merged to lay the foundation for the establishment of the InfiniBand Trade Association, which released the InfiniBand architecture specification 1.0 in 2000.

In 1999, the chip company Mellanox was founded and shortly after its establishment, it allied with NGIO. When NGIO and FIO merged, Mellanox seamlessly transitioned into the InfiniBand ecosystem and launched its first InfiniBand product in 2001.

In 2002, Intel decided to shift towards the development of PCI Express (PCIe, launched in 2004), and Microsoft also withdrew from InfiniBand development.

In 2003, when InfiniBand found a new application domain – computer cluster interconnectivity. In the same year, Virginia Tech constructed a cluster based on InfiniBand technology, securing the third position in the TOP500 list, a global ranking of supercomputers.

In 2004, the Open Fabrics Alliance(OFA) was established, another important non-profit organization related to InfiniBand. OFA and IBTA maintain a collaborative relationship, with IBTA focusing on the development, maintenance, and enhancement of InfiniBand protocol standards, while OFA takes charge of developing and maintaining both the InfiniBand protocol and higher-level application APIs.

In 2005, InfiniBand found another application scenario: connecting storage devices.

By 2009, 181 systems in the TOP500 list had adopted InfiniBand technology.

In 2015, for the first time, InfiniBand technology accounted for more than 50% of the TOP500 list, marking its status as the preferred interconnection technology for supercomputers.

To keep pace with InfiniBand's advancements, Ethernet also underwent continuous development.

In April 2010, IBTA introduced RoCE (RDMA over Converged Ethernet), "porting" RDMA technology from InfiniBand to Ethernet. By 2014, a more mature version, RoCE v2, was proposed. With RoCE v2, Ethernet significantly closed the technological performance gap with InfiniBand, leveraging its cost and compatibility advantages.

In the high-performance networking field, the competition is mainly between InfiniBand and high-speed Ethernet, with both demonstrating considerable strength. Manufacturers with ample resources typically choose InfiniBand, while those prioritizing cost-effectiveness tend to favor high-speed Ethernet.

The Technical Principles of InfiniBand Technology

After tracing the development history of InfiniBand technology, the following text will discuss its working principles, unveiling why it surpasses traditional Ethernet in terms of performance and latency.

Pioneering Advancement: RDMA

As highlighted earlier, a standout feature of InfiniBand is its early integration of the Remote Direct Memory Access (RDMA) protocol. In traditional TCP/IP architectures, data transmission requires the CPU to handle network protocols, such as TCP/IP header operations, retransmission control, and flow management, involving multiple CPU-involved steps and memory replication operations, leading to high latency and significant CPU load.

RDMA allows data to be transferred directly from the user space of the sender to the user space of the receiver without passing through the kernel buffers, thereby reducing intermediate memory copy steps. Additionally, RDMA processes transmission operations through hardware, significantly reducing the demand on the CPU.

By reducing multiple memory copies and system calls, RDMA significantly lowers latency, enabling near-real-time data transmission, which is crucial for HPC and real-time data processing applications.

RDMA's features reduce transaction processing time from milliseconds to microseconds, greatly improving system throughput and response speed. This is one of the technological breakthroughs of the InfiniBand network.

InfiniBand Network Architecture

The network topology structure of InfiniBand is visually represented in the diagram below:

InfiniBand network is built on a channel-based architecture, featuring four primary components:

• HCA (Host Channel Adapter)

• TCA (Target Channel Adapter)

• InfiniBand links (connecting channels, ranging from cables to fibers, and even on-board links)

• InfiniBand switches and routers (integral for networking)

Channel adapters, specifically HCA and TCA, play a crucial role in establishing InfiniBand channels, ensuring both security and adherence to specified Quality of Service (QoS) levels for all transmissions.

Systems leveraging InfiniBand can be structured into multiple subnets, with each subnet capable of supporting over 60,000 nodes. Within a subnet, InfiniBand switches handle layer 2 processing, while routers or bridges facilitate connectivity between subnets.

The second-layer processing in InfiniBand is streamlined. Each InfiniBand subnet is equipped with a subnet manager responsible for generating a 16-bit Local Identifier (LID). InfiniBand switches, comprising multiple ports, facilitate the forwarding of data packets from one port to another based on the LID contained in the Layer 2 Local Routing Header. Notably, switches primarily handle packet management and do not actively generate or consume data packets.

Within the InfiniBand network, data is transmitted in the form of packets, each with a maximum size of 4 KB, utilizing a serial approach.

Leveraging its uncomplicated processing and proprietary Cut-Through technology, InfiniBand achieves a significant reduction in forwarding latency, reaching levels below 100 ns. This latency is notably faster than what traditional Ethernet switches can offer.

InfiniBand Protocol Stack

The InfiniBand protocol embraces a structured layering approach, with each layer functioning independently and delivering services to the layer positioned above it. Please refer to the diagram below for a visual representation:

The Physical layer defines how signals are transmitted over the wire. In an InfiniBand network, this layer includes signal modulation and demodulation, bit synchronization, and error detection. For example, the physical layer transmits digital signals by modulating them into analog signals, which are then sent through copper wires or optical fibers.

The data link layer defines the format of data packets and outlines protocols for essential packet operations like flow control, routing selection, encoding, and decoding in an InfiniBand network. For example, when a switch receives a data packet, it checks the header at the link layer, performs error correction, and then forwards it to the correct output port.

The network layer adds a 40-byte Global Routing Header (GRH) to the data packet for routing, facilitating effective data forwarding.

The transport layer takes charge of delivering the data packet to a designated Queue Pair (QP) and provides instructions to the QP on how to process the packet effectively.

InfiniBand network's well-defined layers collectively constitute a comprehensive network protocol, and its end-to-end flow control forms the bedrock of the network's packet transmission and reception, ensuring lossless networks.

Queue Pairs (QPs) are essential in RDMA technology, consisting of a Send Queue (SQ) and a Receive Queue (RQ). QPs act as basic communication units. When users make API calls to send or receive data, they place the data into the QP. The requests in the QP are then processed in order through polling.

InfiniBand Link Rate

InfiniBand links can be established using either copper cables or fiber optic cables, with dedicated InfiniBand cables chosen based on specific connection requirements.

At the physical layer, InfiniBand defines multiple link speeds, such as 1X, 4X, and 12X, each employing a four-wire serial differential connection, with two wires in each direction.

Over time, InfiniBand network bandwidth has seen continuous upgrades, progressing from SDR, DDR, QDR, FDR, EDR, and HDR to NDR, XDR, and GDR, as depicted in the diagram below:

FS InfiniBand Solutions

FS.com offers a diverse product portfolio covering speeds from 40G to 800G to meet customers' varying speed requirements, including FDR, EDR, HDR, and NDR. Our product line includes InfiniBand Quantum/Quantum-2 switches, InfiniBand modules, InfiniBand adapters, as well as AOC/DAC cables supporting distances from 0.5 meters to 100 meters. These products not only support high-speed interconnects and extremely low latency but also provide scalable solutions.

In addition, FS have 7 local warehouses around the world ensuring fast delivery. FS conducts rigorous performance, reliability, scenario and compatibility testing to ensure product excellence. FS.com has a professional technical team and rich experience in deploying solutions across application scenarios. FS actively provide solutions for high-performance computing, data centers, education, research, biomedicine, finance, energy, autonomous driving, Internet, manufacturing, and telecommunications.

For more details about FS InfiniBand transceivers and cables or how to choose InfiniBand products, you can check these posts:

FS InfiniBand Transceivers and Cables Complete Guide

Tips on Choosing InfiniBand Products for HPC Computing

Conclusion

InfiniBand technology is increasingly favored in high-performance computing and data centers due to its superior low latency, high throughput, and efficient data handling. As a future-proofed technology, InfiniBand technology has continuously innovated and upgraded from SDR, DDR, and QDR to HDR, NDR, and even future 800G. The ongoing development demonstrates its strong scalability and foresight, consistently adapting to and meeting the demands of HPC and data centers.