KLDrive: Fine-Grained 3D Scene Reasoning for Autonomous Driving based on Knowledge Graph
KLDrive addresses fine-grained 3D scene question answering for autonomous driving by coupling an energy-based model for reliable scene knowledge graph construction with a frozen LLM agent that reasons over a constrained symbolic action space. The core insight is that decoupling noisy perception (handled by an EBM that refines multi-source camera and LiDAR detections) from interpretable reasoning (handled by a tool-using LLM with explicit Plan-Execute-Observe loops) substantially reduces hallucinations. The system achieves 65.04\% accuracy on NuScenes-QA and a 46.01 percentage point improvement on counting tasks over prior state-of-the-art, without task-specific fine-tuning of the LLM backbone.
The paper presents a well-motivated hybrid architecture that effectively mitigates hallucinations in driving scene QA through explicit structured reasoning. The two-stage design—energy-based refinement of multi-modal detections followed by constrained LLM agent execution—yields substantial empirical gains, particularly on counting tasks where end-to-end LLMs catastrophically fail (64.46\% vs. 18.45\% for MAPLM). However, the latency of 1.26s per question (57s on edge devices) limits applicability to offline analysis, and the 19-point accuracy gap between predicted (KLDrive-KG) and ground-truth (KLDrive-GT) scene graphs indicates perception remains a significant bottleneck.
The energy-based formulation for scene entity refinement (Equation 3) integrates cross-source corroboration, redundancy penalization, and temporal consistency into a unified objective, providing a principled way to handle noisy multi-modal inputs. The constrained action space design forces the LLM to generate verifiable symbolic traces rather than free-form text, which Section 7.1 attributes to the dramatic reduction in counting errors—from 18.45\% to 64.46\%—demonstrating that explicit tool use effectively eliminates the hallucination-prone direct generation of numeric answers. The system operates without task-specific LLM fine-tuning, using only few-shot in-context exemplars to guide Qwen3-7B.
The perception front-end requires training (Section 6.4 notes the EBM parameters are estimated using binary supervision), making the "no task-specific training" claim in the abstract apply only to the LLM agent rather than the full pipeline. The latency figures (1.26s on A6000, 57.05s on Jetson Orin) confirm the authors' admission that this targets an "offline analysis setting" rather than real-time autonomy. Moreover, the ablation in Table 3 shows that removing the LLM agent drops accuracy to 44.38\%, but it does not isolate which specific EBM energy terms ($E_\mathrm{keep}$, $E_\mathrm{pair}$, $E_\mathrm{attr}$, $E_\mathrm{sup}$) contribute most to the gains, leaving hyperparameter sensitivity and the necessity of the full energy formulation unclear.
The evidence strongly supports the quantitative claims on NuScenes-QA and GVQA, with KLDrive-KG achieving 65.04\% overall accuracy versus MAPLM's 60.17\% (Table 2). The comparison to end-to-end driving LLMs like DriveLM and CREMA is fair, using the same benchmarks and evaluation protocols. However, the KLDrive-GT oracle experiment (84.49\% accuracy) reveals that perception errors still account for roughly 20 points of accuracy loss, suggesting that improvements in the energy model or detection backbones could yield further gains independent of the reasoning module. The paper does not compare against other frozen-LLM tool-use approaches outside the driving domain, which would clarify whether the gains come from the KG representation or the constrained action space paradigm itself.
The paper uses standard public datasets (NuScenes-QA, GVQA) and established detection backbones (RayDN, FocalFormer3D, IS-Fusion), which aids reproducibility. The EBM formulation is mathematically detailed in Section 4.3 (Equations 3-6), though specific hyperparameter values for $\lambda_{dup}$, $\lambda_{tmp}$, $\beta_{tmp}$, etc. are not provided. The code and energy model training procedures are not mentioned in the provided text. The reliance on a frozen Qwen3-7B is clearly specified, but the exact few-shot exemplars and full prompt templates (beyond the partial example in Figure 5) would be needed for exact reproduction.
Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.