MetaRoCE Holds 86% Throughput at 1% Loss Where RoCEv2 Collapses
In this article
Meta has published MetaRoCE, a ground-up redesign of the RDMA transport layer targeting AI training clusters on commodity Ethernet. The core departure from standard RoCEv2 is philosophical: where RoCEv2 assumes the fabric delivers frames in order and uses Priority Flow Control to enforce that assumption, MetaRoCE assumes the fabric is lossy, pushes ordering and path selection entirely into the NIC, and removes PFC and pause frames entirely. For engineers running collective operations across thousands of accelerators, that distinction is not academic — every stalled transfer in an all-reduce or all-to-all holds the entire training step, and at the scale Meta operates (clusters spanning hundreds of thousands of GPUs across multiple data centers), network friction directly strands compute.
The full specification, a DPDK-optimized software reference implementation called libsoftmetaroce, and a compliance test suite are slated to ship at the 2026 OCP Global Summit in October. This is consistent with the broader pattern of software-layer gains outpacing hardware acquisition at the frontier: the bottleneck Meta is attacking is architectural, and the fix is a protocol redesign rather than a new switch ASIC.
Six Design Decisions
Out-of-order delivery as the default. Packets spray across multiple fabric paths and arrive out of order by design. Each packet carries its own destination address, allowing the receiver NIC to write directly to the final memory location — no reorder buffer, no head-of-line blocking. Send operations carry a match token to a posted receive buffer, so a message lands correctly even when earlier messages are still in flight.
Multipathing as a first-class primitive. Each path receives a distinct UDP source port as its ECMP entropy value, which the NIC can change at any point to reroute traffic around a degraded link. Each path maintains its own congestion window and round-trip time estimate, giving the transport enough signal to distinguish packet loss from congestion — a distinction RoCEv2's single-path model cannot make cleanly.
Loss tolerance in place of losslessness. Each path carries its own ordered sequence stream. A gap in a path's 256-bit selective acknowledgment bitvector is interpreted as loss evidence, not reordering, triggering retransmission of exactly the missing packet on the specific path that dropped it. No global go-back-N, no fabric-wide backpressure.
Bilateral congestion control. MetaRoCE combines sender-side ECN-based AIMD with receiver-driven fair-share rate hints: each acknowledgment returns the fraction of inbound bandwidth the receiver allocated to that sender. Meta states that incast resolves in one or two round trips under this scheme.
Topology independence. The only requirements MetaRoCE places on switch hardware are ECN marking and ECMP — both present in virtually every modern data-center switch. It does not require packet trimming, in-network telemetry, credit-based flow control, or switch-side spraying, which means it can operate over fabrics the operator does not fully configure, including multi-tenant cloud environments.
Unified connection state. Standard RDMA scales bandwidth or ordering by multiplying queue pairs, each with a congestion window blind to the others. MetaRoCE separates the concerns: one connection carries many independent ordered streams above and many paths below, governed by a single congestion controller. Connection state no longer grows proportionally with workload parallelism.
Benchmark Results on AMD Hardware
Meta validated MetaRoCE on AMD Pensando programmable NICs, running RCCL collectives on a 64-node AMD GPU cluster against RoCEv2 on both all-reduce and all-to-all operations, delivering higher throughput and lower flow completion times across both. MetaRoCE sustains approximately 86% throughput at a 1% packet loss rate and continues delivering useful bandwidth at 10% loss rather than collapsing. Multiplane validation covered 4-plane and 8-plane topologies with up to 4,000 concurrent connections; throughput scaled linearly with plane count. Simulated plane failures showed traffic redistributing without application involvement or operator intervention.
| Property | RoCEv2 | MetaRoCE |
|---|---|---|
| Fabric assumption | Lossless (PFC enforced) | Lossy (no PFC, no pause frames) |
| Packet ordering | In-order delivery required | Out-of-order by default; direct-write to final memory |
| Multipathing | Discouraged; single path per QP | Native; per-path UDP entropy, window, and RTT estimate |
| Loss recovery | Go-back-N per queue pair | 256-bit selective ACK bitvector; per-path retransmit |
| Congestion control | Sender-side ECN AIMD | ECN AIMD + receiver fair-share rate hints per ACK |
| Switch requirements | PFC, often in-network telemetry | ECN marking and ECMP only |
| Connection state scaling | Grows with QP count | One controller; streams and paths multiplexed |
| Throughput at 1% loss | Degrades significantly | ~86% on 64-node AMD GPU cluster (RCCL) |
| Validated topology | Typically single-plane fat-tree | 4-plane and 8-plane, up to 4,000 concurrent connections |
Open Specification and Deployment Timeline
MetaRoCE extends the multi-vendor ethos that OCP's Ethernet Scalable Unified Network (ESUN) initiative applied to fabric hardware into the transport layer. The three artifacts releasing at the October 2026 OCP Global Summit — the full specification, libsoftmetaroce as the authoritative behavioral reference for silicon development, and the compliance test suite — are designed to let NIC vendors verify conformance independently. Hardware support beyond AMD Pensando is described as underway from additional vendors, though no names are disclosed.
As pipeline architecture becomes the dominant lever for frontier AI gains, the networking layer is being redesigned with the same rigor previously applied to compute. MetaRoCE treats the collective operation — not the individual packet — as the unit of design, and its open release through OCP gives the industry a concrete reference to evaluate and build upon rather than a proprietary implementation to reverse-engineer.