Comparing Performance: InfiniBand EDR vs. 100Gb Ethernet

What Is InfiniBand EDR?

InfiniBand stands out as a pivotal network interconnect technology, widely embraced in supercomputing clusters for its unparalleled reliability, minimal latency, and robust bandwidth capacity. Moreover, propelled by advancements in artificial intelligence, it has emerged as the top choice for interconnecting GPU servers.

What Is Ethernet?

Unlike InfiniBand, the primary goal of Ethernet design is to facilitate smooth information flow among multiple systems. It represents a typical distributed, compatibility-oriented network design. Traditional Ethernet networks are predominantly built using TCP/IP. However, with the maturity of RoCE (RDMA over Converged Ethernet) and iWARP technologies and ecosystems, RDMA has also gained widespread adoption in Ethernet networks. 100Gb Ethernet refers to Ethernet technology that supports data transfer rates of 100 gigabits per second (Gbps). It is a high-speed networking standard designed to meet the increasing bandwidth demands of modern data centers and enterprise networks.

For more information about InfiniBand and Ethernet, please visit: InfiniBand vs. Ethernet: What Are They?

Performance Comparison between InfiniBand EDR and 100G Ethernet

In performance comparison experiments conducted at the University of New Mexico, the performance of IB, RoCE, and Ethernet was evaluated. The tests utilized Mellanox ConnectX-4 EDOR HCA capable of operating in both Ethernet and IB modes. In InfiniBand EDR link layer operation, communication utilized Mellanox MSB7700-ES2F EDR Mellanox IB switches, while 100Gb Juniper QFX5200 data center switches were employed for Ethernet link layer operation. OpenMPI 1.10.3 and OSU MicroBenchmarks were used for performance testing.

When examining latency, InfiniBand exhibited a 10% performance improvement over RoCE for small 8-byte messages. Conversely, native Ethernet displayed considerably higher latency, typically around 10 microseconds. In terms of latency performance, Ethernet operates on a different scale compared to RDMA technologies.

RDMA Protocols: InfiniBand, RoCE, and iWARP

In distributed storage networks, the protocols we use include RoCE, InfiniBand (IE3), and TCP/IP. Among them, RoCE and IB belong to RDMA (Remote Direct Memory Access) technology. Currently, there are three types of RDMA networks, namely InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP.

Among them, InfiniBand is a network specifically designed for RDMA, ensuring reliable transmission at the hardware level with advanced technology but high costs. On the other hand, RoCE and iWARP are both based on Ethernet RDMA technology, enabling high-speed, ultra-low latency, and minimal CPU usage RDMA technology to be deployed on the most widely used Ethernet networks. The RoCE protocol has two versions, RoCEv1 and RoCEv2. RoCEv1 is an RDMA protocol implemented at the Ethernet link layer, while RoCEv2 is implemented at the UDP layer within the Ethernet TCP/IP protocol, addressing scalability issues by introducing IP.

Characteristics of Three RDMA Protocols

InfiniBand: Ensures reliable transmission at the hardware level, offering higher bandwidth and lower latency. However, it comes at a higher cost and requires support for IB network cards and switches.

RoCE: Consumes fewer resources than iWARP and supports more features. It can use regular Ethernet switches but requires network cards that support RoCE.

iWARP: A TCP-based RDMA network that achieves reliable transmission using TCP. Compared to RoCE, in large-scale networks, iWARP's numerous TCP connections may consume a significant amount of memory resources, requiring higher system specifications.

Summary

In conclusion, the comparison between InfiniBand EDR and 100Gb Ethernet highlights the distinct advantages and considerations of each technology in terms of performance, reliability, and cost-effectiveness. While InfiniBand EDR offers superior bandwidth, lower latency, and reliable transmission, it comes with higher costs and requires specialized hardware support. On the other hand, 100Gb Ethernet provides a cost-effective solution with widespread compatibility, although it may exhibit higher latency and lower bandwidth compared to InfiniBand. Ultimately, the choice between the two technologies depends on the specific requirements and priorities of the networking environment, ensuring optimal performance and efficiency for diverse applications.