Infiniband NDR OSFP Solution

The NVIDIA Quantum™-2 InfiniBand platform has emerged as the premier choice for artificial intelligence and large-scale cloud data centers, offering ultra-high bandwidth, ultra-low latency, and simplified operation and maintenance. Quantum™-2 represents a significant leap forward compared to its predecessors, achieving double the port speed, three times the switch port density, five times the switch system capacity, and an impressive 32 times the switch AI acceleration capability. Leveraging the Dragonfly+ topology, the Quantum™-2 network can attain a remarkable 400Gb/s connection capacity for over a million nodes within just three hops, providing AI developers and researchers with unparalleled network performance to address global challenges.

NVIDIA Quantum™-2 InfiniBand Switch

The NVIDIA Quantum™-2 InfiniBand switch comprises two main models: MQM9790-NS2F and MQM9700-NS2F. Their primary distinction lies in the management method, with NVIDIA QM9700 featuring an external management support interface. Both models share identical port types and speeds. The NVIDIA QM9700 series switch supports flexible configurations, accommodating either 64 400G ports or 128 200G ports. Notably, the NVIDIA QM9700 adopts a 1U design with a single panel of 32*OSFP ports, where each OSFP port supports 2X400G speed.

The Quantum™-2 is equipped with the third-generation NVIDIA SHARP technology, ensuring almost infinite scalability for aggregating network data across various scales. With a remarkable 32 times higher AI acceleration capability than its predecessor, the third-generation SHARP technology supports multiple tenants and parallel applications, ensuring optimal performance. Key features such as MPI_Alltoall acceleration, MPI tag matching hardware engine, advanced congestion control, dynamic routing, and self-healing network contribute to significant enhancements for high-performance computing (HPC) and AI clusters.

ConnectX®-7 InfiniBand Card

The NVIDIA ConnectX®-7 InfiniBand network card (HCA) ASIC delivers a staggering data throughput of 400Gb/s, supporting 16 lanes of PCIe 5.0 or PCIe 4.0 host interface. Utilizing advanced SerDes technology with 100Gb/s per lane, the 400Gb/s InfiniBand is achieved through OSFP connectors on both the switch and HCA ports. The OSFP connector on the switch supports two 400Gb/s InfiniBand ports or 200Gb/s InfiniBand ports, while the network card HCA features one 400Gb/s InfiniBand port. The product range includes active and passive copper cables, transceivers, and MPO fiber cables. Notably, despite both using OSFP packaging, there are differences in physical dimensions, with the switch-side OSFP module equipped with heat fins for cooling.

NDR Optical Connection Solution

Addressing the NDR optical connection challenge, the NDR switch ports utilize OSFP with eight channels per interface, each employing 100Gb/s SerDes. This allows for three mainstream connection speed options: 800G to 800G, 800G to 2X400G, and 800G to 4X200G. Additionally, each channel supports downgrade from 100Gb/s to 50Gb/s, facilitating interoperability with previous-generation HDR devices. The 400G NDR series cables and transceivers offer diverse product choices for configuring network switch and adapter systems, focusing on data center lengths of up to 500 meters to accelerate AI computing systems. The various connector types, including passive copper cables (DAC), active optical cables (AOC), and optical modules with jumpers, cater to different transmission distances and bandwidth requirements, ensuring low latency and an extremely low bit error rate for high-bandwidth AI and accelerated computing applications.

OSFP-SR8-800G InfiniBand Transceiver Technology Overview

FS OSFP-SR8-800G stands out as a 2x400Gb/s dual-port multi-mode parallel 8-channel transceiver, employing 100G-PAM4 modulation. This innovative technology utilizes two 4-channel multi-mode MPO-12/APC fiber jumpers, extending its maximum transmission distance to 50 meters. A pivotal feature of this dual-port 2xSR4 transceiver is the incorporation of two internal transceiver engines, facilitating the realization of 64 400Gb/s ports in the Quantum™-2 switch with 32*OSFP ports.

Module Deployment Scenarios

The OSFP-SR8-800G transceiver serves multiple purposes in the NDR InfiniBand Quantum™-2 switch ecosystem, connecting to either other switches or the ConnectX®-7 Adapter/BlueField-3 DPU.

• 1. NDR InfiniBand Quantum™-2 Switch - NDR InfiniBand Quantum™-2 Switch

Utilizing OSFP-SR8-800G transceivers and MPO-12/UPC fiber jumpers, NVIDIA QM9700/QM9790 switches with OSFP ports interconnect at 800G (2x400G) speed within a maximum transmission distance of 50m.

• 2. NDR InfiniBand Quantum™-2 Switch - 400G ConnectX®-7 (OSFP/QSFP/QSFI9112) Adapter

800G to 2x400G links: The dual-port OSFP-SR8-800G transceiver supports up to two 400G ConnectX®-7 adapters and/or DPUs, facilitated by direct-attach multi-mode MPO-12/UPC fiber jumpers. OSFP-SR4-400G-FL optical modules are employed for OSFP ports on the ConnectX®-7 network card side.

• 3. NDR InfiniBand Quantum™-2 Switch - 200G ConnectX®-7 (OSFP/QSFP112) Adapter/BlueField-3 DPU (QSFP112)

When paired with two 1:2 branching MPO fiber jumpers, the dual-port OSFP-SR8-800G transceiver supports up to four 200G ConnectX®-7 adapters and/or DPUs. Similar to the previous scenario, OSFP-SR4-400G-FL optical modules are used for OSFP ports, while QSFP112-SR4-400G optical modules are employed for QSFP112 ports.

In both OSFP-400G-SR4 and Q112-400G-SR4 modules, only two channels within the 400G transceiver are activated, establishing 200G rate links.

• 4. NDR InfiniBand Quantum™-2 Switch - DGX H100 GPU Systems

For DGX-H100 GPU systems featuring 8 "Hopper" H100 GPUs, each dual-port OSFP-SR8-800G transceiver establishes two 400G ConnectX®-7 links from the DGX to the Quantum™-2 switch. This streamlined approach reduces redundancy, complexity, and the number of ConnectX®-7 cards and transceivers compared to the DGX A100 setup. DGX-H100 also supports additional networking options, including up to four ConnectX®-7 and/or two BlueField-3 DPUs for storage I/O using InfiniBand and/or Ethernet, providing versatility for various network configurations.

InfiniBand OSFP-SR4-400G-FL Transceiver

Introduction to Optical Transceiver Technology

FS OSFP-SR-400G-FL stands as a single-port OSFP-packaged SR4 multi-mode parallel transceiver with a speed of 400Gb/s, employing 100G-PAM4 modulation. When coupled with a single 4-channel multi-mode MPO-12/APC fiber jumper, it achieves a maximum transmission distance of 50 meters. When utilized with a 1:2 branching MPO fiber jumper, only two channels on the branching end are activated. This unique configuration automatically creates a 200G speed, effectively reducing power consumption.

Module Usage Overview

The OSFP-400G-SR4 transceiver serves a crucial role in the 400Gb/s ConnectX®-7/OSFP PCIe bus network card, connecting to a single 800Gb/s dual-port 2x400G OSFP transceiver (OSFP-SR8-800G) within the Quantum™-2 InfiniBand switch.

The OSFP-400G-SR4 transceiver supports two distinct speeds, 400Gb/s and 200Gb/s, contingent on the type of fiber jumper used for the connection:

• 400Gb/s mode: Paired with an MPO-12/APC fiber jumper, this mode requires the opposite end's OSFP-SR8-800G transceiver to be paired with two OSFP-SR4-400G-FL transceivers, each connected to 200Gb/s ConnectX®-7/OSFP adapter cards.

• 200Gb/s mode: Utilizing a 1:2 branching MPO fiber jumper, the OSFP-SR4-400G-FL operates at a speed of 200Gb/s (NIDR200), automatically reducing power consumption as only 2 channels are activated. In this mode, the opposite end's OSFP-SR8-800G transceiver can be paired with four OSFP-SR4-400G-FL transceivers, each connected to 200Gb/s ConnectX®-7/OSFP adapter cards.

• 1. NDR InfiniBand Quantum™-2 Switch - 2 x 400G ConnectX®-7 (OSFP) Adapter

The dual-port OSFP-SR8-800G transceiver, when combined with two direct-attached multi-mode MPO-12/APC fiber jumpers, can support a maximum of two ConnectX®-7/OSFP adapters. Each adapter employs an OSFP-SR4-400G-FL transceiver, achieving a maximum transmission distance of 50 meters.

• 2. NDR InfiniBand Quantum™-2 Switch - 4x200G ConnectX®-7 (OSFP) Adapter

Similarly, the dual-port OSFP-SR8-800G transceiver, in conjunction with two direct-attach multi-mode MPO-12/APC fiber jumpers (M4MPOA12FB), can support a maximum of two ConnectX®-7/OSFP adapters. Each adapter utilizes an OSFP-SR4-400G-FL transceiver, achieving a maximum transmission distance of 50 meters. The ConnectX®-7/OSFP adapter operates with OSFP-SR4-400G-FL modules with only two channels activated, establishing a 200G rate link and automatically reducing power consumption.

MFA7U10-H030 InfiniBand Cables

Active Optical Cable (AOC) Technology Overview

FS NVIDIA InfiniBand MFA7U10-H030 compatible OSFP 400G to QSFP56 2x200G breakout cable is an active optical splitter cable (AOC) converting OSFP to 2x QSFP56, enabling a speed of 400Gb/s to 2x 200Gb/s. This cable connects each pair of data signals from 8 multi-mode fiber (MMF) pairs originating from a single OSFP end to four pairs on each QSFP56 multi-port end.

AOC Usage Insights

The MFA7U10-H030 serves the purpose of connecting an NDR switch with OSFP ports to 2 HDR switches or HCA QSFP56 ports.

• NDR InfiniBand Quantum™-2 Switch - 2x HDR InfiniBand Quantum Switch

• NDR InfiniBand Quantum™-2 Switch - 2 x 200G QSFP56 Connect®-6 Adapter/BlueField-2 DPU

MTP Fiber Optic Patch Cable

This MTP cable is an 8-fiber passive multi-mode fiber optic patch cable, featuring 8 individual fibers with 4 fibers in each direction. The MTP connectors, of the 8-degree angled polished (APC) type, minimize optical signal reflections and ensure optimal signal integrity. This cable is designed to interconnect two switches or connect switches to two network adapters, supporting pluggable fiber optic 400GbE/NDR transceivers.

The MTP cable is crafted for linking two switches or establishing connections between switches and two network adapters. This cable is versatile and can be paired with pluggable fiber optic 400GbE/NDR transceivers. For instance, it seamlessly integrates with the OSFP 800G 2xSR4 dual-port transceiver designed for InfiniBand and Ethernet systems on the switch end. Additionally, it complements the OSFP-400G-SR4 or QSFP112-400G-SR4 transceivers when used in ConnectX®-7 network adapters and BlueField-3 DPUs.

Introduction to FS Testing

The comprehensive testing includes hardware compatibility testing (plug/unplug, port restart, power restart), software compatibility testing (connectivity, shutdown/no shutdown, parameter testing), and performance testing (DDM, bit error rate, stability testing). The link has been operational at full speed for one week, exhibiting no packet loss, errors, CRC errors, or interruptions. All tested parameters align with relevant industry standards.

Summary

With a deep understanding and extensive implementation experience in building high-performance InfiniBand networks, FS offers optimal combinations of switches, intelligent network cards, and AOC/DAC/optical modules tailored to diverse application scenarios. Our products and solutions bring substantial advantages and value to data centers, high-performance computing, edge computing, artificial intelligence, and other applications, enhancing business acceleration capabilities with a blend of low cost and outstanding performance.