Remote Direct Memory Access (RDMA)

What Is Remote Direct Memory Access (RDMA)?

Remote Direct Memory Access (RDMA) is a technology that allows two networked computers to exchange data in their main memory without using the processor, cache, or operating system of either computer. Like locally-based Direct Memory Access (DMA), RDMA frees up system resources, improving throughput and performance. This results in faster data transfer rates and lower latency between RDMA-enabled systems. RDMA benefits both networking and storage applications.

The concept of zero-copy networking is central to RDMA. It enables direct data reading from the main memory of one computer and writing to the main memory of another. This bypasses the kernel networking stack in both computers, enhancing network performance.

Consequently, communications between the two systems are completed much faster than in comparable non-RDMA networked systems.

RDMA has proven useful in applications that require fast, large-scale parallel high-performance computing (HPC) clusters and data center networks. It is beneficial for big data analysis, supercomputing environments that process applications, and machine learning tasks requiring low latency and high transfer rates. Additionally, RDMA is used between nodes in compute clusters and for latency-sensitive database workloads.

Overall, RDMA serves as a powerful tool for enhancing data transfer performance in specialized applications. To better understand how RDMA achieves these improvements, it is important to look at the specific steps and mechanisms involved in its operation.

How Does RDMA Work?

RDMA allows more direct and efficient data movement into and out of a server by implementing a transport protocol in the network interface card (NIC) of each communicating device. For instance, two networked computers can be equipped with NICs that support the RDMA over Converged Ethernet (RoCE) protocol, allowing them to communicate using RoCE.

With Remote Direct Memory Access (RDMA), the host application initially places its data into the memory buffer and then calls the network or storage driver. The RDMA stack allows the hardware adapter to access the original buffer directly. This bypasses much of the traditional software stack, such as TCP/IP, and avoids numerous memory buffer copy operations.

As a result, latency is reduced, and the data is transmitted as quickly as possible. If the device on the other end also implements RDMA, the entire data transfer between the two systems can be completed much faster compared to systems that do not use RDMA.

Advantages of RDMA

Using RDMA offers several significant advantages:

• Zero-copy: Applications can transfer data directly to and from the buffers without involving the network software stack, avoiding copying data between network layers.

• Kernel bypass: Data transfers can occur directly from userspace, eliminating the need for context switches to the kernel.

• No CPU involvement: Remote memory can be accessed without CPU resources on the remote machine, which means the remote CPU's caches are not filled with the accessed memory content, and the remote process is not interrupted.

• Message-based transactions: Data is managed as discrete messages rather than a continuous stream. This simplifies the application's task of separating the stream into different messages or transactions.

• Scatter/gather entries support: RDMA naturally works with multiple scatter/gather entries. It can read multiple memory buffers and send them as one stream, or receive one stream and write it to multiple memory buffers.