Multi-Board Compute Fabrics: PCIe vs Serial RapidIO for Mission Subsystems

Why multi-board systems exist

A nontrivial signal-processing or fusion subsystem often needs more compute than a single processor board provides. The work fans out across multiple boards — perhaps a Kintex-class FPGA for the front end, a multi-core DSP for the algorithmic middle, and a general-purpose CPU board for control and storage. Those boards have to talk to each other at high bandwidth and low latency.

For the past 15 years, two backplane fabrics have dominated this niche: PCI Express (PCIe) and Serial RapidIO (SRIO). They look superficially similar — both are point-to-point serial fabrics, both run at multi-gigabit lane rates, both support switched topologies — but they have different operational characters that matter at design time.

PCIe: the commercial standard

PCIe is the fabric everyone already has. Every modern CPU, every modern FPGA with a transceiver, every off-the-shelf SSD or NIC supports PCIe natively. This translates to:

• Cheap silicon and tooling. PCIe IP cores, switches, retimers, and protocol analyzers are commodity items.
• Mature software stacks. Linux has been driving PCIe devices for two decades; PCIe is the default device interface on x86 and ARM.
• Single-root tree topology. A PCIe system has one root (typically a host CPU) and a tree of devices beneath it. Devices do not normally talk to each other; all traffic goes through the root.

The tree topology is PCIe's blessing for software simplicity and its curse for multi-board mission systems. If three FPGA boards in a system need to exchange high-bandwidth data with each other (typical of a radar back-end that distributes range bins across boards for parallel processing), every transfer goes through the host CPU's memory controller. This serializes the system at the host and wastes its DRAM bandwidth on traffic that has nothing to do with the host.

Modern PCIe extensions — peer-to-peer DMA, NTB (non-transparent bridging) — partially work around the tree limitation, but each adds complexity and is only well-supported on subsets of silicon.

SRIO: the embedded standard

Serial RapidIO was designed from the start for peer-to-peer fabrics in embedded systems. Its operational character differs:

• No host concept. Every endpoint is symmetric. Any endpoint can DMA to any other endpoint without involving a host.
• Low protocol overhead. SRIO has smaller packet headers than PCIe and a simpler flow-control model, giving slightly higher useful bandwidth on the same link rate.
• Better worst-case latency. SRIO's per-hop latency is roughly half PCIe's for the same data, primarily because of the simpler header processing.
• Limited silicon support. SRIO is mostly found in multi-core DSP families and some FPGAs. It is not native to mainstream x86 or ARM processors.

For a system whose data movement is intrinsically peer-to-peer — a multi-DSP signal processing system, a multi-FPGA radar back-end — SRIO is architecturally a better fit. For a system that is host-centric and primarily moves data between a host and devices, PCIe is the better fit.

Decision framework

A few questions decide which fabric is right:

Is the data flow host-centric or peer-to-peer? If most transfers are between a host CPU and a device (NIC, SSD, GPU), PCIe is correct. If most transfers are between devices (FPGAs talking to DSPs talking to other DSPs), SRIO is correct.

What's the silicon constraint? If the chosen processors and FPGAs all have PCIe natively but no SRIO, designing in SRIO requires bridges, which add cost and latency. The pragmatic answer for any system built around modern x86 or ARM is PCIe even if SRIO would be architecturally cleaner.

What's the maintenance posture? PCIe's commodity tooling and broad expert pool means a sustainment engineer at a forward depot is more likely to be able to diagnose a fabric issue. SRIO debug requires more specialized knowledge and tooling.

What's the latency budget? For applications with sub-10 µs end-to-end latency requirements across multiple hops, SRIO's lower per-hop latency starts to matter. For applications with 100+ µs budgets, the difference is in the noise.

Sustainment relevance

A sustainment platform's compute footprint is primarily host-centric — application servers, decision engines, databases. PCIe is the right fabric for internal systems.

The relevance of this discussion is that the edge systems we integrate with (radar back-ends, signal-processing platforms, multi-DSP sensor processors) are often SRIO-based. When the platform ingests data from those systems, the interface is usually a high-level network protocol (Ethernet, IPv4) at the edge of the SRIO domain, not direct SRIO peering. Understanding what's on the other side of that interface lets us reason about latency budgets, message rates, and graceful degradation.