← All insights
ISR7 min read

Sub-Millisecond H.264 Pipelines on Zynq for ISR Downlink

An SoC-FPGA design pattern for capture-to-radio H.264 video with end-to-end latency under one millisecond, suitable for ISR platforms where decision loops cannot tolerate frame buffering.

The problem

A loitering ISR platform — fixed-wing UAS, tethered aerostat, or maritime ASV — produces high-rate H.264 video that has to traverse a contested datalink and reach an operator station with enough timeliness for the operator's decision loop. Off-the-shelf H.264 encoders on application processors typically pipeline 4–8 frames through their hardware accelerator and add another 30–80 ms of memory copies and buffering before the bitstream hits the radio. At 30 fps that is a 250–400 ms glass-to-glass delay, which is fine for situational awareness but unacceptable for closed-loop guidance, sensor cueing, or stabilization assistance.

The architecture below collapses encode-and-decode latency to under one millisecond per side by moving the entire pixel path into FPGA fabric and eliminating the host-memory round trip.

Architecture

The reference platform is a Zynq-class SoC FPGA — the processing system (PS) hosts a thin Linux instance for telemetry and configuration, while the programmable logic (PL) carries the real-time pixel pipeline:

  • Capture front-end: a parallel-port or MIPI CSI-2 receiver in fabric latches pixels as they arrive from the sensor, with no DMA into PS DRAM. A short line-buffer (8–16 lines) is the only buffering between sensor and codec.
  • Encoder core: H.264 baseline profile with a fixed slice structure, no in-loop deblocking, and intra-refresh in lieu of full I-frames. The entire encode pipeline is purely combinatorial within each slice; the slice itself emits at line-rate.
  • Network adaptation: the bitstream goes directly to a custom RTP/UDP packetizer in fabric, which writes Ethernet frames to a hardened MAC. PS is not in the data path.
  • Decoder side: a mirror-image structure on the ground station — Ethernet → RTP depacketizer → H.264 decoder → display sink — also entirely in PL, with the host CPU only handling control plane.

End-to-end measured latency on a 1080p30 link, including PHY serialization on a 1 Gb Ethernet hop, is consistently under 950 µs. The variance is dominated by the network MAC and is bounded by the slice boundary.

Why this matters for sustainment

Two operational profiles benefit:

  1. Remote diagnostics for forward-deployed maintainers. A maintainer wearing a head-mounted camera streams to a rear depot SME during component troubleshooting. Conversational round-trip time stays below the human-perceptual threshold even over a constrained satcom hop, which keeps the SME's verbal cueing actually useful instead of stale.
  2. Cross-echelon sustainment awareness. Repair stations push live video of contested wreckage or contested-supply staging to higher-echelon planners. Sub-millisecond latency means the operator's pan / zoom inputs are responsive enough to perform real reconnaissance rather than "stale slideshow" review.

Implementation notes

  • Slice height is the dominant tunable knob. We default to 8 macroblock rows (128 px at 1080p) — small enough to fit one slice per fabric clock burst, large enough that header overhead stays under 4%.
  • Intra-refresh is essential. Periodic full I-frames create transient bitrate spikes that the contested-link MAC can't always absorb. Spreading I-blocks across a refresh cycle smooths the rate.
  • The Linux side must not touch the framebuffer. Any mmap of the capture region from userland reintroduces PS-side DRAM traffic and breaks the latency bound.
  • Forward error correction lives in the network layer, not the encoder. We deliberately avoid the temptation to put FEC inside the slice; doing so couples loss recovery to frame structure and makes adaptive bitrate harder.

Power and SWaP

A Zynq-7045 implementation of this pipeline measures roughly 6.8 W under continuous 1080p30 encode, including the MAC and the thin PS instance. A 7035 is sufficient for 720p; a Kintex-class part is needed for 4K. SWaP at the 720p tier comfortably fits a small UAS payload bay.

What we'd hand a programmer

A complete reference build of this pipeline is straightforward to scope as a 6–9 month engineering deliverable: a board support package, a fabric IP stack with vendor-licensed H.264 cores, a Linux userspace control plane, and a closed-loop latency test rig. Lodestar's role in such a procurement is to integrate this pipeline into the broader sustainment decision graph — not to re-implement the codec.