Revolutionizing InfiniBand Networking for HPC Clusters

High-performance computing (HPC) clusters require robust networking solutions to handle the immense data processing demands of scientific research, simulations, and data analysis. InfiniBand technology has emerged as a powerful solution, offering unparalleled high-speed networking capabilities. This article explores the potential of InfiniBand in unleashing the true power of HPC clusters and showcases the InfiniBand products provided by FS.

One of the key aspects of InfiniBand is its diverse range of link rates at the physical layer. Each link consists of a four-wire serial differential connection (two wires in each direction).

These link rates, such as Single Data Rate (SDR), Double Data Rate (DDR), Quad Data Rate (QDR), and more, offer varying levels of raw signal bandwidth and data bandwidth. Over time, the network bandwidth of InfiniBand has evolved significantly, introducing faster and more efficient link rates, including the latest Next Data Rate (NDR), which provides a staggering 100 Gb/s (400 Gb/s for 4X link rates). The following diagram illustrates the progression of InfiniBand's network bandwidth, with the speed based on 4X link rates.

InfiniBand networking utilizes a cost-intensive approach called Fat Tree topology for achieving lossless communication between compute nodes. This topology, depicted in the diagram below (squares represent switches and ellipses represent compute nodes.), comprises two layers: the core layer, dedicated to traffic forwarding, and the access layer, connecting compute nodes.

Implementing a Fat Tree in InfiniBand networks incurs high costs. For instance, if a convergence switch has 36 ports, half of them (18 ports) must connect to compute nodes, while the remaining half connects to the core switches in the upper layer.