Why is east-west traffic the real AI cluster bottleneck

There is a gap between what GPU vendors report in training benchmarks and what operators actually see when they run production inference workloads. MFU looks compelling on paper. Then you push mixture-of-experts or disaggregated prefill and decode across a real cluster, and the number drops faster than it should.

The instinct is to blame the hardware. The hardware is usually fine.

The topology is the problem

Modern inference workloads are fundamentally about coordination. Tensor parallelism splits work across accelerators and then reassembles it. Mixture-of-experts routes tokens to specialists and aggregates results. Disaggregated prefill and decode separate compute phases across different hardware. All of these patterns generate all-to-all traffic.

All-to-all traffic on a static fabric is where p99 latency numbers become real. When every node needs to talk to every other node and the physical paths inside and between racks are fixed cables, contention is structural. You can tune it. You cannot eliminate it.

The dead zone between racks

At 800G, passive copper DAC reach has already shrunk below 3 meters due to the physics of PAM4 signaling at 112 Gbps per lane. Inter-rack distances of 3 to 7 meters fall into a dead zone where AEC cables with DSP retimers are the current best answer. Every AEC hop adds 50 to 100 nanoseconds of latency from FEC overhead. That compounds across a large cluster running all-to-all patterns.

At 1.6T, the dead zone gets worse. Passive copper reach shrinks further. Nothing in the current stack makes those links programmable or reconfigurable.

Predictable latency under contention is the metric that matters

The industry has spent enormous energy optimizing peak bandwidth. That is the wrong optimization target for inference. What matters is how latency behaves under load, specifically at the tail. A static fabric gives you good p50 latency and poor p99 because contention events are bursty and the topology has no flexibility to absorb them.

A software-defined physical layer can absorb them. When the connectivity inside and between racks is wireless and reconfigurable in under a millisecond, you adapt paths to the current traffic pattern rather than routing around a fixed layout.

That is what Rivvor is building. The demonstration is at the OCP Global Summit in October 2026.