English

NVIDIA Quantum-2 InfiniBand Technology FAQs

Posted on Feb 29, 2024 by
1.0k

With the rapid development of technology, the demand for HPC in fields continues to grow. Addressing this demand, NVIDIA Quantum-2 Infiniband technology has emerged as a solution. This technology offers high-speed and low-latency data transfer and processing capabilities, providing exceptional performance for distributed computing. In this article, we will delve into the features and applications of NVIDIA Quantum-2 Infiniband technology, and address some FAQs about this technology.

What Is NVIDIA Quantum-2 Infiniband?

NVIDIA Quantum-2 Infiniband utilizes the seventh-generation NVIDIA InfiniBand architecture to provide HPC developers and researchers with superior network performance and rich features to help them solve challenging problems. It delivers software-defined networking, network computing, performance isolation, advanced acceleration engines, remote direct memory access (RDMA), and blazing-fast speeds of up to 400 Gb/s for advanced supercomputing data centers.

Overall, NVIDIA Quantum-2 is a 400Gbps NDR InfiniBand networking platform consisting of Mellanox InfiniBand Quantum-2 switches, ConnectX-7 network adapters, BlueField-3 data processing units (DPUs), 400G NDR InfiniBand transceivers and all the software to support the new architecture.

FS Products under the Technology 

The table below shows the Mellanox InfiniBand switch, IB network adapters and IB transceivers from FS under NVIDIA Quantum-2 Infiniband Technology.

Type
Product
Features
Mellanox InfiniBand NDR Switch
64 x NDR 400G, 32 OSFP Connectors, Managed
64 x NDR 400G, 32 OSFP Connectors, Unmanaged
InfiniBand Network adapter
Single-Port QSFP112, PCIe 5.0 x 16, Secure Boot
Single-Port OSFP, PCIe 5.0 x 16, Crypto and Secure Boot
Single-Port OSFP, PCIe 5.0 x 16, Secure Boot
Single-Port OSFP, PCIe 5.0 x 16
400G NDR InfiniBand Transceivers
OSFP 400G SR4 PAM4, 850nm, 50m
OSFP 400G DR4 PAM4, 1310nm, 500m

FAQs about NVIDIA Quantum-2 Infiniband Technology

Tech FAQs

Q1: Can UFM be used to monitor RoCE networks?

A1: No, UFM (Unified Fabric Manager) only supports InfiniBand.

Q2: At what cluster scale do you recommend using UFM?

A2: It is recommended to configure UFM for all IB networks. UFM provides not only OpenSM but also other powerful management and interface functionalities. Moreover, whenever customers have network management requirements, it is recommended to configure UFM.

Mellanox InfiniBand Switch FAQs

Q1: In the Super Pod networking setup, if the last SU has fewer than 32 nodes, let's say only 16 nodes, can the Leaf switches in the last SU be reduced to just 4? This would result in two network interfaces of the same node connecting to one Leaf switch. Will there be any issues with the SHARP tree?

A1: It is possible, but not recommended. Mellanox InfiniBand NDR switches can support 64 SAT (SHARP Aggregation Tree).

SHARP

Q2: On an InfiniBand NDR switch, can one port be connected to an NDR cable and the other port be connected to a one-to-two NDR 200 cable?

A2: Yes, it is possible. The port splitting configuration for NDR interfaces must be done on the switch side.

Q3: The InfiniBand NDR switch has 64 400Gb ports. Why is it said to have 32 QSFP56 ports?

A3: Due to size and power constraints, a 2U panel can only accommodate 32 cages. Each cage can support two 400Gb ports, resulting in 64 ports. It's important to differentiate between the cage and port concepts when referring to NDR switches and their interfaces.

Q4: Is UFM's functionality the same for managed NDR switches and non-managed NDR switches?

A4: Yes, the functionality is the same.

400G NDR InfiniBand Module FAQs

Q1: Can different modules on each end be connected with a cable to establish data connectivity? For example, if the server uses OSFP and the switch uses QSFP112.

A1: In the case of InfiniBand, you must use InfiniBand modules and cables. For InfiniBand switches, the ports are OSFP. Both ends should use the recommended modules from NVIDIA to ensure compatibility and support.

Q2: If NDR is compatible with HDR and EDR, is this kind of cable and module only available in one piece??

A2: Yes, there is no separate solution for cables and modules.

Q3: Should the modules on the OSFP NIC side be flat?

A3: Yes, the NIC should use flat modules.

Q4: 400G IB and 400G Ethernet, except for the optical modules, are the cables the same?

A4: The optical cables are the same. Note that they are the APC type with an 8-degree bevel angle.

InfiniBand Network Adapter FAQs

Q1: Is RDMA not supported in Ethernet mode for the InfiniBand network adapter?

A1: RDMA over Ethernet (RoCE) is supported. It is recommended to use the Nvidia Spectrum-X solution.

Q2: Is there any mention of simplex or duplex for IB network adapters?

A2: All IB network adapters support duplex communication. The concept of simplex or duplex is irrelevant for current devices because the physical channels for sending and receiving data are already separate.

Conclusion

Adhering to Quantum-2's networking guidelines ensures efficient communication and resource sharing. By configuring and bonding network interfaces appropriately, users can further enhance system performance. NVIDIA Quantum-2 Infiniband technology plays a vital role in technological advancements and will continue to drive progress in high-performance computing.

You might be interested in

Knowledge
Knowledge
Knowledge
See profile for Sheldon.
Sheldon
Decoding OLT, ONU, ONT, and ODN in PON Network
Mar 14, 2023
399.7k
Knowledge
See profile for Irving.
Irving
What's the Difference? Hub vs Switch vs Router
Dec 17, 2021
372.3k
Knowledge
See profile for Sheldon.
Sheldon
What Is SFP Port of Gigabit Switch?
Jan 6, 2023
346.6k
Knowledge
See profile for Migelle.
Migelle
PoE vs PoE+ vs PoE++ Switch: How to Choose?
May 30, 2024
429.2k
Knowledge
Knowledge
Knowledge
Knowledge
See profile for Moris.
Moris
How Much Do You Know About Power Cord Types?
Sep 29, 2021
305.8k