An In-Depth Look at NVIDIA NDR 400G InfiniBand

The relentless demand for faster data processing and transfer speeds in high-performance computing (HPC) and cloud computing has led to groundbreaking innovations in networking technology. At the forefront is NVIDIA Mellanox with its NDR InfiniBand(IB), bringing an impressive 400Gb/s capability. This article delves into the InfiniBand technology standard, its key components, and the performance enhancements delivered by this stellar advancement.

InfiniBand Architecture

InfiniBand employs two queue pair extraction techniques that allow applications to directly transfer data from the adapter to application memory (referred to as RDMA) and vice versa. Compared to operations in the TCP/IP protocol, IB utilizes Single Data Copy (SDP) to directly copy data from the network card to the user's application space, bypassing involvement of the core memory. This zero-copy approach achieves maximum protocol throughput during large-scale data processing. The InfiniBand protocol adopts a layered structure, with each layer being independent and providing services to the layer above it. The physical layer defines how bit signals form symbols on the wire and then how symbols form frames, data symbols, and data padding. It provides detailed specifications for constructing efficient data packets and signaling protocols. You can read the article InfiniBand Network and Architecture Overview to learn more.

InfiniBand Basic Components

The InfiniBand network topology consists of four categories of components:

• Host Channel Adapter (HCA): It acts as a bridge between the memory controller and the Target Channel Adapter (TCA).

• Target Channel Adapter (TCA): It packages digital signals from I/O devices (e.g., network cards, SCSI controllers) and sends them to the HCA.

• InfiniBand links: They are the fiber connections between HCAs and TCAs. The IB architecture allows hardware vendors to connect TCAs and HCAs using 1, 4, or 12 fiber cables.

• Switches and routers: Both HCAs and TCAs essentially act as host adapters and feature programmable Direct Memory Access (DMA) engines with certain protection capabilities.

Rate Evolution of InfiniBand

In high-concurrency and high-performance computing applications where customers demand exceptional bandwidth and low latency, IB networking emerges as a suitable solution. Currently, InfiniBand supports various bandwidth modes, and future plans include even higher modes. The following table presents the bandwidth modes using 4X as an example:

Overview of NVIDIA NDR 400Gb/s InfiniBand

Comparisons with Previous InfiniBand Generations

When comparing NDR 400Gb/s InfiniBand with previous generations such as EDR and HDR, it becomes evident that the former offers substantial advantages. The increased bandwidth and reduced latency of NDR 400Gb/s IB outperform its predecessors, enabling faster and more efficient data transfers. Moreover, compatibility and interoperability considerations ensure a smooth transition for organizations already utilizing IB technology.

NVIDIA Mellanox NDR 400Gb/s InfiniBand boosts performance in multiple ways:

• 1. Higher Bandwidth: It delivers faster data transfer, improving communication efficiency in HPC clusters.

• 2. Reduced Latency: It minimizes communication delays, enabling rapid data transmission and responsiveness.

• 3. Enhanced Scalability: It supports large-scale HPC environments, handling increased data volumes and facilitating parallel processing across clusters.

• 4. Improved Efficiency: NDR IB optimizes data transfer, minimizing overhead and maximizing network resource utilization for higher system efficiency and application performance.

• 5. Advanced Features: It introduces enhanced error detection, congestion control, and supports advanced network topologies, ensuring better reliability and stability.

• 6. Future-Proofing: NDR IB's high data transfer rate accommodates growing workloads and emerging technologies without the need for extensive infrastructure upgrades, providing a technology roadmap for future HPC requirements.

NVIDIA 400G InfiniBand Products Available from FS

Conclusion

NVIDIA NDR 400G InfiniBand technology sets a new benchmark for high-speed networking. Its performance enhancements, including increased bandwidth, reduced latency, scalability, and reliable data transmission, empower organizations across industries to achieve new levels of productivity and efficiency, driving the next generation of computing.