A Quick Look at the Differences: RoCE vs Infiniband RDMA vs TCP/IP

As we all know, a network protocol is a set of rules that govern data transmission. Remote Direct Memory Access (RDMA) and TCP/IP networking protocols are commonly used in distributed storage networks. Both RoCE and Infiniband are RDMA technologies, so what is the difference between them and TCP/IP? What are the differences between RoCE and Infiniband RDMA? This article explains RoCE vs Infiniband RDMA vs TCP/IP in detail.

Understanding RoCE, Infiniband RDMA, and TCP/IP

RoCE

RDMA over Converged Ethernet, or RoCE for short, is a network protocol defined by the InfiniBand Trade Association (IBTA) standard, which allows data to be transferred from one machine to another, reducing the operating load on the CPU. Generally, there are two main versions of RoCE: RoCEv1 and RoCEv2. RoCE v1 is an Ethernet link layer protocol, while RoCE v2 is an Internet layer protocol. RoCE achieves lower latency than its predecessor, the IWARP protocol. Compared with the InfiniBand RDMA structure, RoCE's advantages, such as high cost performance, high performance, and low power consumption, can bring many benefits to modern data centers.

Infiniband RDMA

InfiniBand (IB) is a network specially designed for RDMA, which guarantees reliable transmission at the hardware level, but the cost is relatively high. InfiniBand RDMA has extremely high throughput and extremely low latency and is used for computer-to-computer data interconnection.

Different from RoCE, Infiniband RMDA implements data transmission through Infiniband adapters or switches instead of Ethernet. Therefore, Infiniband can achieve lower latency and higher bandwidth than RoCE. A specific type of Ethernet switch has port-to-port latency of 230 ns, while an InfiniBand switch with the same number of ports has 100 ns.

TCP/IP

TCP/TP, or Transmission Control Protocol/Internet Protocol, is used to interconnect network devices over the Internet. It identifies how data should be packetized, addressed, transmitted, routed, and received. TCP/IP places lots of emphasis on accurate data transmission between two computers. If the system encounters some problem while sending the message in one go, the entire message must be sent again.

In addition, the functionality of TCP/IP divides into four different layers: datalink layer, internet layer, transport layer, and application layer. Data must go through these four layers before being received at the other end. Then, TCP/IP will reassemble the data by passing the layers in the opposite order and present it to the receivers. In this way, you can improve the performance or security of data centers by upgrading certain layers rather than the whole system.

Network Protocol Evolution: From TCP/IP to Infiniband RDMA to RoCE

TCP/IP to RDMA

Traditional data centers usually use Ethernet technology to form a multi-hop symmetrical network architecture, and use the TCP/IP network protocol stack for transmission. However, compared with RoCE and Infiniband RDMA, TCP/IP is gradually not suitable for high-performance computing business demands. Its main limitations are as follows:

• Tens of microseconds of delay caused by TCP/IP protocol stack processing

• TCP protocol stack processing causes high server CPU load

To reduce network latency and CPU occupancy, the RDMA function is generated on the server side, and finally achieves the effects of high bandwidth, low latency, and low resource occupancy.

Infiniband RDMA to RoCE

Both InfiniBand RDMA and RoCE can implement remote memory access network protocols. The two currently have their own advantages and disadvantages on the market, and both are used in HPC cluster architecture or large-scale data centers.

Comparing the two, InfiniBand has better performance. But InfiniBand is a dedicated network technology. It cannot inherit the user's accumulation and platform of operation on the IP network, which causes the high cost in operation and maintenance. Therefore, carrying RDMA based on the traditional Ethernet network is also inevitable for the large-scale application of RDMA. To guarantee RDMA performance and network layer communication, many network switches use RoCEv2 to carry high-performance distributed applications.

 

     ''Also Check- RDMA over Converged Ethernet Guide

 

Finding the Difference Between RDMA and TCP/IP

As mentioned above, network protocols have evolved in three ways. The three networks also have their own advantages and disadvantages, and they can be selected as appropriate when facing different application scenarios. The following table lists the several differences between RoCE, Infiniband RDMA, and TCP/IP.

• High Scalability: All these three network protocols have high scalability and flexibility, with Infiniband being the most scalable. A single subnet of Infiniband can support tens of thousands of nodes. Besides, it also provides relatively simple and scalable architecture, creating almost unlimited cluster sizes through Infiniband routers.

• High performance: Since TCP/IP burdens CPU processing resources and latency, it performs worse compared to the other two network protocols. And RoCE increases speed and power in enterprise data centers while reducing total cost of ownership without replacing Ethernet infrastructure. As for Infiniband, it uses serial links and buses to send data one bit at a time, enabling faster, more efficient communications.

• Easy management: Although RoCE and Infiniband RDMA have lower latency and higher performance than TCP/IP, the latter is easier to deploy and manage. The network administrators who use TCP/IP to build up devices and network connectivity only need little central management.

• Cost-efficiency: For enterprise data centers with limited budget, Infiniband is probably not a good choice. It uses expensive IB switching ports to carry a large number of applications, increasing the computing cost, maintenance cost, and management cost of enterprises. In contrast, RoCE and TCP/IP using ethernet switches are more cost-effective. Therefore, Infiniband switches are more popular in HPC data centers.

• Network devices: As the table shows above, both RoCE and TCP/IP achieve data transmission through ethernet switches, while Infiniband uses IB switches with independent architecture to carry applications. Typically, IB switches must be interconnected with devices that support IB protocol and are relatively closed and difficult to replace.

 

     ''Also Check- Why HPC Data Centers Need InfiniBand Interconnection

 

Which One Is the Best for Data Center?

Nowadays, data center networks require maximum bandwidth and extremely low latency from the underlying interconnect. Under these circumstances, the traditional TCP/IP network protocol cannot keep pace with data center requirements since they burden CPU processing resources and high latency.

For those enterprises deciding between RoCE and Infiniband RDMA, their unique requirements and cost should be taken into consideration. If they prefer the most high-performance network connection, Infiniband is much better. While for those looking for the best performance, easy management and limited cost, then they can choose RoCE for their data centers.