UAS Obstacle Avoidance: Fusing mmWave Radar with Monocular Vision

Why fuse radar and vision

A small unmanned aircraft system operating autonomously in degraded visibility — dust, smoke, fog, low-light, GPS-denied indoor environments — cannot rely on optical sensing alone. Cameras have well-known failure modes: featureless surfaces defeat optical flow, low contrast defeats stereo, motion blur from rapid maneuvering defeats both.

Adding millimeter-wave radar gives the platform a sensor with complementary failure modes. Radar sees through dust, smoke, and most fog. It provides direct range and range-rate measurements without the computational cost of stereo or structure-from-motion. It is unaffected by lighting.

What radar does not give you is semantics. A radar return tells you "there is an object at 12 m, closing at 3 m/s." It does not tell you whether the object is a tree branch (minor course correction), a power line (mission-ending), or another aircraft (regulatory event). Vision, when it works, provides that interpretation.

Architecture

The reference compute stack is a three-tier heterogeneous SoC:

• DSP for the radar signal-processing chain: dechirp, range/Doppler FFTs, CFAR detection, target list generation.
• FPGA for the camera pipeline: image undistortion, optical flow, region-of-interest extraction.
• ARM for the fusion layer, mission planner, and flight-control I/O.

Each task has a different computational character. The radar chain is regular, predictable, latency-critical — a DSP workload. The camera pipeline is per-pixel, parallel, can absorb arbitrary parallelism — an FPGA workload. The fusion layer is irregular, branchy, easier to develop in C++ — an ARM workload.

mmWave radar ──→ DSP ──┐
                       │
                       ├──→ ARM fusion ──→ Flight controller
                       │
Camera ──→ FPGA ───────┘

The fusion problem

Radar gives you a target list — typically a few tens of detections per frame, each annotated with range, range-rate, azimuth, elevation. Vision gives you detected objects in the image plane, each with bounding box and class confidence.

Two common patterns for cross-sensor association:

Geometric association. Project each radar detection into the image plane using the known sensor-to-sensor calibration. For each radar detection, find the camera detection whose bounding box best contains the projected point. Simple, but sensitive to calibration error — a 1° rotation error produces a 17 cm projection error at 10 m range, which can move the projected point outside its bounding box.

Probabilistic gating. Each detection becomes a Gaussian in a joint state space (position, velocity, class). The Mahalanobis distance between radar and camera detections is computed in that joint space, with cross-sensor weighting tuned to reflect each sensor's uncertainty. Handles calibration error gracefully but requires honest characterization of sensor noise.

For a fielded autonomy system, gating is the right answer. The calibration error is real and not constant — the rigid mount flexes thermally and mechanically — and a fusion algorithm that breaks at 1° rotation error is too brittle for an aerial platform.

Failure-mode design

The interesting design questions are how the autonomy degrades when one sensor fails:

• Camera blinded (smoke, dust, sun in lens). Radar still produces a target list. Fall back to radar-only obstacle avoidance with a conservative buffer (classification is lost). Essentially "stop or slow down for anything that's there, regardless of what it is."
• Radar saturated (heavy clutter, multipath in a structured environment). Vision still works. Fall back to vision-only with reduced top speed, since vision-only obstacle detection has higher latency and lower range accuracy than radar.
• Both degraded (heavy rain at low altitude, dense smoke). The platform should refuse to make autonomous maneuvering decisions and either hold position (if it can determine it has clear airspace immediately around it) or initiate a controlled descent. The "do not pretend you can see when you can't" rule.

These failure modes need to be designed in from the start. A fusion algorithm that assumes both sensors are always available will produce nonsense outputs (with high confidence) when one fails.

Sustainment relevance

The DoD interest in small autonomous platforms — ISR, comms relay, contested last-mile delivery — is well documented. From a sustainment perspective, the same architectural pattern serves two roles:

1. Tactical resupply autonomy. A small fixed-wing or rotary platform carrying a 5–20 kg payload needs reliable obstacle avoidance to traverse contested terrain. The compute stack above is the substrate.
2. Inspection of supply infrastructure. A small platform performing autonomous patrols of fuel lines, perimeter fences, or vehicle parks needs the same sensing capability for safe navigation around poorly-mapped structures.

Both use cases benefit from on-device fusion. Both fail if the system depends on a real-time link to a ground station for perception.