Differences Between InfiniBand and Ethernet NICs: A Selection Guide

As data centres and high-performance computing environments grow increasingly complex, the choice between InfiniBand and Ethernet technologies can significantly impact overall efficiency and effectiveness. Each technology offers unique advantages and is suited to specific application scenarios. This article will delve into the main differences between InfiniBand and Ethernet network cards, offering a comprehensive selection guide to help you make an informed decision based on your specific needs.

What are InfiniBand and Ethernet NICs？

InfiniBand NICs

The InfiniBand network interface card(NIC) is a type of network adaptor used in high-performance computing and data centres. The InfiniBand adapter, also known as the Host Channel Adapter (HCA), is the crucial connection point where an InfiniBand end node, such as a server or storage device, links to the InfiniBand network. InfiniBand is a high-speed, low-latency interconnect technology designed to support applications such as high-performance computing, large-scale data transfer, and clustered computing. By providing fast data transfer and low-latency communication, IB NICs can be used to connect servers, storage devices, and other network equipment to achieve high performance and high throughput data transmission.

Ethernet NICs

The Ethernet NIC is a network adaptor that is inserted into a motherboard slot and supports Ethernet protocol standards. Each network adaptor has a globally unique physical address known as a MAC address. Using the MAC address, data can be accurately sent to the destination computer. There are many types of Ethernet NICs, and they can be classified in various ways. e.g. bandwidth, network interface, and bus interface.

For a detailed introduction to IB NICs and Ethernet NICs, Also Check：Ethernet Card vs. IB Card

Comparison of InfiniBand and Ethernet NICs

IB NICs and Ethernet NICs offer unique advantages tailored to different networking requirements. IB NICs are designed for high performance, providing higher bandwidth and lower latency. Ethernet NICs, on the other hand, are versatile and scalable, suitable for various network scales and types. Understanding the differences between IB NICs and Ethernet NICs can help make informed decisions based on specific network performance requirements.

Technology and Protocols

IB NICs use the InfiniBand protocol, designed primarily for high-performance computing (HPC) environments. They employ Remote Direct Memory Access (RDMA) technology, enabling ultra-low latency data transfers. In contrast, Ethernet NICs use widely adopted Ethernet protocol standards, such as TCP/IP. These NICs are suitable for various network applications, including enterprise and home networks. While Ethernet NICs are simple to use, they exhibit higher latency when handling large amounts of data.

Performance Characteristics

IB NICs and Ethernet NICs differ significantly in bandwidth, latency, and throughput. IB NICs offer bandwidth options ranging from 40Gbps to 400Gbps, with latency typically in the microsecond range. This makes them ideal for HPC and scientific research applications requiring rapid data processing. Ethernet NICs provide bandwidth options from 1Gbps to 100Gbps, with emerging standards reaching up to 400Gbps. However, they usually have higher latency, ranging from hundreds of microseconds to milliseconds. Nevertheless, Ethernet NICs perform well for most routine data transfer tasks.

Use Cases and Applications

IB NICs are mainly used in HPC environments, scientific research, large-scale simulations, and financial services. These applications require high message rates and low latency, such as artificial intelligence and big data analytics. On the other hand, Ethernet NICs are widely used in various scenarios, including enterprise networks, cloud computing, web hosting, and home networks. Their strong compatibility and flexibility make them the preferred choice for many networking solutions.

Latency and Reliability

IB NICs benefit from RDMA technology, allowing data packets to be forwarded without involving the CPU, significantly reducing packet processing latency (typically 600ns for send and receive). Ethernet NICs, based on TCP or UDP, have send and receive latencies of around 10us, resulting in a considerable difference. Additionally, IB NICs achieve lossless transmission through end-to-end flow control, minimizing latency jitter and providing a highly reliable network environment. In contrast, Ethernet can suffer from buffer congestion and packet loss in extreme conditions, impacting the stability of data transmission performance.

Overall, InfiniBand and Ethernet network cards each have their own strengths and weaknesses, making them suitable for different use cases. When choosing the right network card, it is essential to consider the specific network environment requirements and budget comprehensively to optimise network performance and improve work efficiency.

Choosing the NIC for the Right Solution

FS meet the requirements of various network environments, whether it be enterprise networks, data centres, or high-performance computing (HPC) environments, providing you with flexible and scalable solutions.

H100 InfiniBand Solution

FS H100 InfiniBand solution, built on the NVIDIA® H100 GPU and integrated with PicOS® software and AmpCon™ management platform, is designed to align with the network topology of HPC architecture. The H100 architecture incorporates the IB NICs. The ConnectX-7 InfiniBand card is a groundbreaking product in the ConnectX series of industry-leading network adapters. It features a PCIe 5.0 x16 host interface and offers 400Gb/s single-port transmission. RDMA delivers low latency and high performance, capable of handling 330 million to 370 million messages per second. This intelligent interconnect is suitable for computing and storage platforms based on x86, Power, Arm, GPU, and FPGA, making it the highest performance and most flexible solution to meet the growing demands of data centre applications.

Learn more about H100 InfiniBand Solution：NVIDIA® InfiniBand H100 Network

RoCE Network Solution

RoCE Network Solution offers a cutting-edge, lossless network by supporting PFC, ECN, and efficient RoCEv2 traffic forwarding, which minimizes GPU idle time. The RoCE Network Solution utilises the NVIDIA® ConnectX®-7 MCX75310AAC-NEAT, featuring a single port with 400Gb connectivity and support for both InfiniBand and Ethernet protocols. It enables a maximum transmission unit (MTU) ranging from 256 bytes to 4KB and supports messages up to 2GB. For storage workloads, the MCX75310AAC-NEAT offers a comprehensive set of software-defined, hardware-accelerated networking, storage, and security capabilities, allowing organisations to modernise and secure their IT infrastructures efficiently.

Learn more about RoCE Network Solution: Empower HPC with RoCE Network

Conclusion

When selecting network interface cards, it's essential to weigh the specific application scenarios and requirements. Ethernet NICs, due to their cost-effectiveness, compatibility, and wide applicability, are suitable for enterprise networks and data centres. In contrast, IB NICs play a vital role in high-performance computing and scientific research due to their ultra-high bandwidth, ultra-low latency, and high reliability. By understanding the working principles, performance metrics, and application scenarios of Ethernet and IB NICs, enterprises and research institutions can make wiser decisions to optimise network performance and enhance work efficiency.