Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms
Vision-language models are moving onto battery-powered edge devices, yet their total-system energy footprint on NPU-equipped single-board computers has remained largely unmeasured. This paper presents a hardware-instrumented study of Qwen2-VL-2B-Instruct inference on a Raspberry Pi 5 with a Hailo-10H NPU and a Jetson Orin Nano Super, separating fixed request energy from per-token decode costs. It shows that NPU offloading cuts Pi energy per token by roughly two-thirds, that text generation dominates vision encoding at typical response lengths, and that the Pi-NPU and Orin-GPU stacks have comparable per-token energy with a crossover near 40 output tokens. The findings are practically useful for engineers choosing deployment stacks under power and thermal constraints.
The paper delivers a carefully instrumented, statistically grounded characterization of VLM inference energy on two distinct edge platforms. Its strongest contribution is the linear decomposition of request energy into fixed and per-token components via forced-output-length regression ($R^2 \ge 0.9997$). However, the core hardware comparison is confounded by differing quantization formats (W4A8 HEF vs. Q4_K_M GGUF) and serving stacks, and the key stack-crossover estimate rests on a single prompt-image pair per resolution. The authors are transparent about these limits, but the headline claims should be read as stack-level deployment guidance rather than silicon verdicts.
The measurement protocol is rigorous: an INA228 hardware energy accumulator with validity gates, paired $t$-tests with Holm correction, and a modality ablation (LLM-T / VLM-T / VLM-V) that cleanly isolates the cost of adding an image. The fixed-length probe provides a compelling regression-based separation of intercept and slope, showing that decode energy surpasses fixed image energy beyond roughly 2--12 tokens. The thermal findings are also robust: the Pi CPU backend throttles in every 30-minute session while the NPU backend stays $\approx 30^\circ$C cooler with no throttle flags.
The study's generality is sharply limited by a single model family (Qwen2-VL-2B), one hardware unit per platform, and a mostly single-day measurement campaign.
The forced-output crossover analysis uses only one prompt and one image at each of three resolutions, so the reported 38--44 token crossover is an extrapolation from $3 \times 7 = 21$ trials per stack rather than a broad sample. The quality--energy sweep is explicitly labeled exploratory, but its single-run per setting and 4-point accuracy steps make the reported minima at 672 px and 64 tokens unreliable.
The evidence solidly backs the Pi-specific claims---the 65.5% energy-per-token reduction and thermal headroom are supported by hundreds of trials and paired statistics---and the decode-dominance result aligns with Zhan et al.'s finding that decoding accounts for 86--97% of total energy on GPU-class hardware. By contrast, the NPU--GPU crossover claim is weaker: the prompt-paired per-token difference is not significant ($p=0.12$), and the crossover rests on a narrow probe.
The authors provide unusually precise hyperparameters, software commits (llama.cpp b9412, HailoRT 4.21.0), and INA228 settings, but they do not release code, prompts, or raw data. They also caution that absolute Joules are uncalibrated against a traceable load, and the Hailo GenAI serving path requires proprietary compiled binaries, so an independent lab must rebuild the stack from scratch and accept shifted absolute numbers.
Vision-language models (VLMs) are moving onto edge devices, but little is known about how much energy they draw on single-board computers with NPU or GPU accelerators, or where that energy goes. We measure total-system energy, latency, and thermal behavior of Qwen2-VL-2B-Instruct and its text-only sibling on a Raspberry Pi 5 with a Hailo-10H NPU and on a Jetson Orin Nano Super, on both CPU and accelerator, and separate the fixed energy of each request from the energy that grows with output length. Because the accelerators use different quantization and serving software, we compare deployment stacks rather than silicon. Three findings stand out. First, NPU offloading makes the Pi a much more efficient VLM platform: it cuts energy per generated token by about two-thirds, leaves the host CPU largely free, and avoids the thermal throttling that the CPU backend reaches in sustained use. Second, an image adds both fixed and per-token energy, but at typical response lengths generating the answer costs far more than processing the image. Third, the Jetson GPU favors short responses through its lower fixed cost per request, whereas the Pi NPU favors longer ones through its lower cost per output token; their energy curves cross near 40 output tokens.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.