InfiniBand Architecture Specification Frequently Asked Questions

InfiniBand has found widespread adoption in high-performance computing (HPC) environments, data centers, and other settings that require rapid and reliable data communication. This article answers frequently asked questions about InfiniBand, covering various aspects such as the concept of InfiniBand, its significance, applications, etc.

1: What is InfiniBand?

InfiniBand is an industry-standard, channel-based, switched fabric interconnect architecture for server and storage connectivity. This interconnect technology employs a switched fabric topology, where devices are connected through a network of switches rather than traditional bus architecture. This allows for more scalable and flexible configurations, as well as improved fault tolerance and overall system reliability. It is widely used in high-performance computing (HPC) environments, data centers, and other settings where fast and reliable data communication is essential for optimal system performance.

For more information: Getting to Know About InfiniBand

2: Why is InfiniBand Important?

High-performance applications, such as biomedical research, drug discovery, data mining, fluid dynamics, and weather analysis, demand efficient message passing and I/O capabilities to expedite the computation and storage of extensive datasets.

For AI workloads, particularly those involving large and intricate models, the computational demands are intensive. To streamline model training and handle extensive datasets, AI professionals are increasingly adopting distributed computing. This involves distributing workloads across interconnected servers or nodes connected through high-speed, low-latency network connections.

Additionally, enterprise applications, including databases, virtualization, network services, and critical vertical markets like financial services, insurance, and retail, necessitate computing systems with optimal performance.

InfiniBand technology has been instrumental in powering large-scale supercomputing deployments for complex distributed scientific computing and has emerged as the practical network for training large, intricate models. With its ultra-low latency, InfiniBand has become a vital accelerator for today's mainstream scientific computing and artificial intelligence applications. InfiniBand interconnect solutions, combined with servers, integrated multi-core processors, and accelerated computing storage, deliver peak performance to address these challenges.

3: What Performance Range Is Offered by InfiniBand?

InfiniBand offers a multi-tiered approach to link performance, featuring diverse speeds such as 56/40G FDR, 100G EDR, 200G HDR, 400G NDR, and 800G NDR. Each of these link speeds not only ensures low-latency network communication but also delivers superior overall throughput. This positions InfiniBand as an optimal choice for data center I/O interconnectivity.

4: How Many Nodes Does InfiniBand Support?

The InfiniBand Architecture can accommodate tens of thousands of nodes within a single subnet. The actual scalability of an InfiniBand network is influenced by factors such as the switch design, the network topology (e.g., fat-tree, hypercube), and the specific InfiniBand specifications implemented in the hardware. It's common for InfiniBand networks to support hundreds or thousands of nodes in practice, making them a robust choice for high-performance and large-scale computing environments.

5: What Type of Cabling Does InfiniBand Support?

InfiniBand supports various types of cabling, with the choice depending on the specific InfiniBand generation or specification being used. The most common types of InfiniBand cabling products include:

• InfiniBand DAC cables are used for short-distance in-rack & neighbor-rack cabling, with a maximum speed reaching up to 800G. These cables are cost-effective and suitable for high-speed, low-latency communication over relatively short distances.

• InfiniBand AOC cables are used for longer-distance connections up to 100 meters and offer the advantages of reduced weight and better signal integrity over extended distances.

• Infiniband Transceivers are suitable for fiber links up to 10km and scenarios requiring higher performance.

6: What Is RDMA over Converged Ethernet (RoCE) and How Does It Relate to InfiniBand?

RoCE is a network protocol that enables the use of RDMA over Layer 2 and Layer 3 Ethernet networks.

RDMA, a technology enabling direct data transfers between the memory, GPU, and storage of remote systems without CPU involvement, supports swift and low-latency data transfers across networks. In contrast to traditional networks involving multi-step data transfers, where data moves from the source system's memory through the network stack, over the network, and eventually into the destination system's memory, RDMA streamlines this process, significantly enhancing data transfer efficiency.

While initially embraced in the High-Performance Computing (HPC) sector through InfiniBand, RDMA is now being leveraged in enterprise Ethernet through RoCE. Today, with the adoption of GPU computing and large-scale AI use cases within cloud environments, Ethernet can be a practical solution when it is running RoCE.

7: Does InfiniBand Support Software Defined Networking (SDN)?

InfiniBand can support Software Defined Networking (SDN). SDN is an architectural approach to networking that separates the control plane (where network decisions are made) from the data plane (where traffic is forwarded). InfiniBand, with its flexibility and adaptability, can be integrated into SDN architectures to provide programmability, automation, and centralized control over network resources.

8: What Is the Relationship between InfiniBand and Fibre Channel or Ethernet?

The InfiniBand architecture complements Fibre Channel and Ethernet, offering superior performance and enhanced I/O efficiency. Positioned uniquely, InfiniBand emerges as the preferred I/O interconnect, progressively replacing Fibre Channel in many data centers. Ethernet connects seamlessly into the edge of the InfiniBand fabric and benefits from better access to InfiniBand architecture-enabled compute resources. This facilitates IT managers in achieving a more balanced allocation of I/O and processing resources within the InfiniBand structure.

Related Articles:

How to Choose Storage Protocol: Fiber Channel vs InfiniBand

InfiniBand vs. Ethernet: What Are They?

9: What Will It Take to Integrate InfiniBand Architecture into a Virtualized Data Center?

The growth of servers based on multi-core CPUs and the utilization of multiple virtual machines are driving the demand for more I/O connections per physical server. For instance, a typical VMware ESX server environment requires multiple Ethernet NICs and Fibre Channel HBAs, leading to increased I/O costs, wiring complexity, and management intricacies.

InfiniBand I/O virtualization addresses these challenges by providing unified I/O across the computing server landscape, significantly enhancing virtual machines' LAN and SAN performance. It allows effective isolation of computing, LAN, and SAN domains for independent scalability of resources. The result is a more adaptable virtual infrastructure.

In VMware ESX environments, virtual machines, applications, and vCenter-based infrastructure management run on familiar NIC and HBA interfaces. This enables IT managers to seamlessly leverage the mentioned value propositions while minimizing disruption and learning curves.

InfiniBand optimizes data center productivity in enterprise vertical applications such as customer relationship management, databases, financial services, insurance services, retail, virtualization, cloud computing, and web services. InfiniBand-based servers offer data center IT managers a unique combination of performance and energy efficiency, forming a hardware platform that provides the highest productivity, flexibility, scalability, and reliability to optimize TCO.

How FS Can Help

FS can provide high-performance InfiniBand solutions through a comprehensive product portfolio, including InfiniBand transceivers and cables with speeds of 800G, 400G, 200G, 100G, and 56/40G, NVIDIA InfiniBand adapters and NVIDIA InfiniBand switches. From cutting-edge R&D to global warehouses, FS's professional team is poised to deliver customized and effective solutions based on your needs, guaranteeing network stability and optimal performance.