NVIDIA Quantum-2 InfiniBand Technology FAQs

With the rapid development of technology, the demand for HPC in fields like artificial intelligence continues to grow. Addressing this demand, NVIDIA Quantum-2 Infiniband technology has emerged as a solution. This technology offers high-speed and low-latency data transfer and processing capabilities, providing exceptional performance for distributed computing. In this article, we will delve into the features and applications of NVIDIA Quantum-2 Infiniband technology, and address some FAQs about this technology.

NVIDIA Quantum-2 Infiniband utilizes the seventh-generation NVIDIA InfiniBand architecture to provide AI developers and researchers with superior network performance and rich features to help them solve challenging problems. It delivers software-defined networking, network computing, performance isolation, advanced acceleration engines, remote direct memory access (RDMA), and blazing-fast speeds of up to 400 Gb/s for advanced supercomputing data centers.

Overall, NVIDIA Quantum-2 is a 400Gbps NDR InfiniBand networking platform consisting of Mellanox InfiniBand Quantum-2 switches, ConnectX-7 network adapters, BlueField-3 data processing units (DPUs), 400G NDR InfiniBand transceivers and all the software to support the new architecture.

The table below shows the Mellanox InfiniBand switch, IB network adapters and IB transceivers from FS under NVIDIA Quantum-2 Infiniband Technology.

Q1: Can UFM be used to monitor RoCE networks?

A1: No, UFM (Unified Fabric Manager) only supports InfiniBand.

Q2: At what cluster scale do you recommend using UFM?

A2: It is recommended to configure UFM for all IB networks. UFM provides not only OpenSM but also other powerful management and interface functionalities. Moreover, whenever customers have network management requirements, it is recommended to configure UFM.

Q1: In the Super Pod networking setup, if the last SU has fewer than 32 nodes, let's say only 16 nodes, can the Leaf switches in the last SU be reduced to just 4? This would result in two network interfaces of the same node connecting to one Leaf switch. Will there be any issues with the SHARP tree?

A1: It is possible, but not recommended. Mellanox InfiniBand NDR switches can support 64 SAT (SHARP Aggregation Tree).

Q2: On an InfiniBand NDR switch, can one port be connected to an NDR cable and the other port be connected to a one-to-two NDR 200 cable?

A2: Yes, it is possible. The port splitting configuration for NDR interfaces must be done on the switch side.

Q3: The InfiniBand NDR switch has 64 400Gb ports. Why is it said to have 32 QSFP56 ports?

A3: Due to size and power constraints, a 2U panel can only accommodate 32 cages. Each cage can support two 400Gb ports, resulting in 64 ports. It's important to differentiate between the cage and port concepts when referring to NDR switches and their interfaces.

Q4: Is UFM's functionality the same for managed NDR switches and non-managed NDR switches?

A4: Yes, the functionality is the same.

400G NDR InfiniBand Module FAQs

Q1: Can different modules on each end be connected with a cable to establish data connectivity? For example, if the server uses OSFP and the switch uses QSFP112.

A1: In the case of InfiniBand, you must use InfiniBand modules and cables. For InfiniBand switches, the ports are OSFP. Both ends should use the recommended modules from NVIDIA to ensure compatibility and support.

Q2: If NDR is compatible with HDR and EDR, is this kind of cable and module only available in one piece??

A2: Yes, there is no separate solution for cables and modules.

Q3: Should the modules on the OSFP NIC side be flat?

A3: Yes, the NIC should use flat modules.

Q4: 400G IB and 400G Ethernet, except for the optical modules, are the cables the same?

A4: The optical cables are the same. Note that they are the APC type with an 8-degree bevel angle.

InfiniBand Network Adapter FAQs

Q1: Is RDMA not supported in Ethernet mode for the InfiniBand network adapter?

A1: RDMA over Ethernet (RoCE) is supported. It is recommended to use the Nvidia Spectrum-X solution.

Q2: Is there any mention of simplex or duplex for IB network adapters?

A2: All IB network adapters support duplex communication. The concept of simplex or duplex is irrelevant for current devices because the physical channels for sending and receiving data are already separate.

Conclusion

Adhering to Quantum-2's networking guidelines ensures efficient communication and resource sharing. By configuring and bonding network interfaces appropriately, users can further enhance system performance. NVIDIA Quantum-2 Infiniband technology plays a vital role in technological advancements and will continue to drive progress in high-performance computing.