On-Device Inference at the Forward Edge: Algorithms, Power, and Practical Limits

Where this conversation usually goes wrong

A common pattern in defense AI discussions is to size an inference accelerator by the TOPS number on the datasheet and assume that "more TOPS = more capability." That framing collapses a real engineering trade-off into a single scalar and produces architectural decisions that look reasonable on a slide deck but break in the field.

A useful counter-frame: a 6 TOPS INT8 accelerator running at sustained load draws roughly 5–8 W. A 60 TOPS accelerator running at the same fraction of peak draws 30–60 W. The difference is the entire SWaP budget of a man-pack. If the workload doesn't actually need the larger device, you've spent the budget on capability you can't use.

What the smaller class is good at

A 6–12 TOPS edge accelerator, integrated with an ARM application processor, video decode hardware, and 4–8 GB of LPDDR memory, runs the following workloads well in real time:

• Eight to sixteen simultaneous 1080p video streams at 15 fps each, with one to two AI tasks per stream. Typical task mix: a YOLOv5-class object detector plus a lightweight classifier or tracker.
• Standard perimeter analytics — intrusion detection, loitering detection, perimeter line crossing, license-plate recognition, basic action recognition (sitting, standing, walking, running). Each task at the 15–30 fps tier per channel.
• Pre-trained models from the standard model zoo — MobileNet, EfficientNet, YOLOv5/v6/v7, ResNet50, ASR models in the wav2vec-base family. Anything in this regime runs at real-time latency.
• A modest amount of on-device retraining or adaptation — fine-tuning the last few layers of a pre-trained backbone on a couple of hundred local examples. Full from-scratch training does not happen on these devices; that's a depot or cloud workload.

What it doesn't do well

Three workload classes look feasible on paper but fail in practice:

• Real-time inference on large language models or large multimodal models. Even a 7B-parameter LLM in 4-bit quantization is on the edge of memory-feasibility for an 8 GB device, and the per-token latency is measured in hundreds of milliseconds. For workloads that genuinely need a generalist model, the accelerator is the wrong device class. Either run the LLM at the depot and accept the round-trip, or use a much larger compute class.
• Per-pixel dense prediction at native resolution and high frame rate. Semantic segmentation at 1080p30 — for every pixel, classify what kind of terrain or object — is borderline. It works for one or two channels; it does not scale to the eight-channel video board mentioned earlier.
• Anything that requires sustained 100% utilization in a thermally-constrained enclosure. The accelerator's quoted TOPS number assumes silicon at 50 °C. A fanless industrial enclosure at 60 °C ambient with the accelerator at 95 % utilization will thermally throttle within ten to twenty minutes. Sustained workloads should be sized at 50–60 % of peak, not at peak.

A deployment pattern that works

For forward-base perimeter, distributed depot security, or unmanned ground vehicle situational awareness, a workable architectural pattern is:

1. Distribute the inference, centralize the curation. Each edge node runs its own perimeter analytics, locally. False-positive review and model retraining happen at the depot, batched. The link between edge and depot is intermittent and asymmetric — high-rate upload of confidence events, low-rate download of updated model weights.
2. Quantize aggressively for the production model, full-precision for the validation model. Production runs in INT8 because that's what the accelerator likes. Validation, retraining, and adversarial-example generation run in FP16 or FP32 at the depot. The two models stay version-locked.
3. Treat the model registry as part of the sustainment supply chain. Each model version is a logistics item: it has a version, a validation report, a list of platforms it's been tested on, and a deployment path that's the same as any other software supply. Models that bypass this registry — pulled from an engineer's laptop, trained ad hoc — should not run in production.

Power budgeting

A useful first-order budget for a man-pack edge inference unit:

That is the budget you actually need to plan around. A 100 Wh battery — typical for a man-pack — gives you 5–8 hours at this draw. If the application demands longer, the answer is either a bigger battery (SWaP penalty) or a duty-cycled compute mode (algorithmic work).

Sustainment relevance

The applicability to sustainment is twofold:

• Asset condition awareness in degraded comms. An edge inference node monitors a piece of equipment locally, only escalating to the depot when its on-device classifier flags an anomaly. This means the comms link is not a single point of failure — the sustainment posture degrades gracefully rather than going blind.
• Distributed cueing. A perimeter camera that recognizes an unauthorized vehicle can cue a sustainment-control node directly, without waiting for human-in-the-loop confirmation, when the consequence of a false positive is low (e.g., redirecting a robotic sentry, not authorizing weapons release).

Both patterns require the same architectural property: the inference happens at the device, and the network is used for control flow and supervisory loops, not for the data plane.