English
RoCE
What Is RoCE?
RDMA over Converged Ethernet (RoCE) is a network protocol that enables Remote Direct Memory Access (RDMA) over Ethernet networks. By bypassing the CPU and enabling direct memory access between servers, RoCE delivers high throughput and low latency for data-intensive applications, making it a critical technology in modern data centers, high-performance computing (HPC), and AI workloads.
Types of RoCE
Generally, there are two RDMA over Converged Ethernet versions: RoCE v1 and RoCE v2. It depends on the network adapter or card used.
RoCE v1
The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
RoCE v2
The RoCE v2 protocol overcomes the limitation of version 1 being bounded to a single broadcast domain (VLAN). By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to a network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). As RoCEv2 fully supersedes the original RoCE protocol, references to the RoCE protocol generally denote the RoCE v2 protocol, unless explicitly specified as the first generation of RoCE.
Benefits and limitations of RoCE
Benefits
Since RoCE has direct access to memory data via network interface rather than through the kernel, it can enable low-latency and high-performance transmission.
Low Latency and High Throughput: RoCE minimizes data transfer delays by bypassing traditional TCP/IP stack processing, making it ideal for latency-sensitive applications like AI training and real-time analytics.
Efficient CPU Utilization: RDMA reduces CPU overhead, freeing processing power for application workloads, which enhances overall system efficiency.
Scalability in Ethernet Networks: RoCE enables RDMA functionality over standard Ethernet infrastructure, supporting cost-effective scaling in modern data centers.
Seamless Integration with HPC and AI: RoCE facilitates high-speed data exchanges required for distributed computing in HPC and AI, ensuring optimized training and inferencing workloads.
Limitations
Due to the performance bottlenecks of traditional Ethernet networks, general RoCE applications still suffer from performance losses such as congestion, packet loss, and latency jitter in high-performance businesses.
Congestion: In multi-cast scenarios, queues can become congested, with queue latency that cannot be ignored.
Packet loss: Compared to FC, traditional Ethernet is prone to congestion and packet loss, with the resending of lost packets easily leading to data disorder.
Latency jitter: The Ethernet network experiences large amounts of jitter, and its store-and-forward mode leads to complex lookup processes and high forwarding latency.
How RoCE Works
RoCE operates over Ethernet by leveraging RDMA functionality to enable zero-copy data transfer. This process involves:
Direct Memory Access: Data is moved directly between the memory of two servers without CPU involvement.
Packetization: Data is encapsulated in RoCE packets and sent over Ethernet.
Hardware Acceleration: Specialized network adapters, such as NVIDIA ConnectX, implement RoCE for maximum performance.
Applications of RoCE
High-Performance Computing (HPC): RoCE is essential in HPC clusters, providing fast interconnects for parallel processing and complex simulations.
AI and Machine Learning: AI training and inferencing rely on low-latency data transfer, where RoCE ensures seamless communication between GPU servers and storage systems.
Storage Networks: RoCE optimizes protocols like NVMe-oF (NVMe over Fabrics), delivering high-speed, low-latency storage access in data centers.
Financial Services: Real-time trading and analytics systems benefit from RoCE’s ultra-low latency and high throughput.
Videos
01:11
Jun 26, 2024
354
0
Global Delivery Service | FS
Related Topics
Solutions
PicOS® for H100 InfiniBand Solution
Based on the NVIDIA® H100 GPU, along with PicOS® software and AmpCon™ management platform, the FS H100 Infiniband solution is tailored according to the network topology of HPC architecture, including infiniband network, management network, and storage net
Empower HPC Network with 400G RoCE Lossless Solution
This solution offers a perfect 400G interconnect solution for QSFP-DD switches and OSFP network cards, addressing the compatibility issues between different port encapsulations. And it is tailored according to the network topology of HPC architecture, inc
PicOS® for Video Streaming Solution
FS integrates advanced switches, PicOS®, and AmpCon™ management platform to design a modern data center architecture for 4K/8K video service providers. It enables efficient network management, enhanced capacity and reliability, and reduced overall Opex.
-