LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding
LED-BENCH targets a realistic but under-evaluated setting: answering engineering questions from complete, large-format drawings in mechanical, architectural, and transportation domains. The benchmark contains 107 sheets and 377 expert-verified questions that require models to localize fine-grained evidence and integrate information across distant regions or views. Its central finding is that current multimodal models remain far from solving this problem—the best evaluated model achieves only 52.65% accuracy, and performance consistently degrades on harder reasoning tasks.
The paper makes a timely contribution by focusing on full-sheet, multidisciplinary engineering-drawing understanding rather than isolated crops or single-view tasks. Its strongest assets are the expert-led construction protocol and the diagnostic intervention study, which together pinpoint where models fail. That said, the benchmark is small—107 images and 377 questions—and the reliance on proprietary models with unspecified or provider-default hyperparameters weakens reproducibility.
The multidisciplinary scope and expert-led pipeline are genuine differentiators: questions are grounded in real drawing-reading needs, difficulty levels are reviewed by engineers, and the failure-stage taxonomy precisely pinpoints why models struggle. The supplementary visual-access interventions are also valuable, showing that even simple fixed-crop augmentation can raise domain-average accuracy by nearly 13 percentage points, which strongly supports the authors' thesis that fine-grained visual access matters.
The dataset is very small compared to contemporaries: 107 images and 377 questions versus MechVQA's 3,281 images and 20,778 tasks, and the architecture subset is especially thin with only 21 drawings and 75 questions. Run-to-run stability is concerning for certain models, with GPT-5.6 Luna showing a ±3.03 point standard deviation across three runs and Gemini-3.5 Flash varying by over 12 points on architectural questions, suggesting that aggregate leaderboards may be noisy. Finally, the visual-access interventions are not an apples-to-apples comparison: the Locate then reread and Search then reread methods consume three API calls per question versus one for the baseline, conflating improved visual access with increased inference budget.
The evidence broadly supports the central claims: the accuracy gap between Medium and Hard questions is consistent across all 11 models, and the diagnostic error analysis aligns with the observed gains from crop-based interventions. Comparisons to prior work in Table 1 are factually presented, though the authors should acknowledge that benchmarks with many more samples may be more robust to overfitting and easier to use for iterative model development, whereas LED-BENCH functions more as a targeted stress test.
The authors publish a GitHub link and provide exact scoring formulas, prompts, and aggregation logic in Appendix B, which is commendable. However, reproduction is hampered by the use of eight proprietary models accessed through unnamed API gateways with provider-default temperature and sampling settings; no seeds are reported, and run-level variance in Appendix D reveals non-negligible jitter. The benchmark does not appear to include a fully open-source baseline implementation or training code, and the drawing collection, while sourced from public materials, would benefit from explicit licensing documentation to ensure redistribution safety.
Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-grained understanding of engineering information and complex reasoning over objects, relations, and evidence distributed across different regions or views. Existing benchmarks cover a range of engineering-drawing understanding tasks involving local regions, individual views, specific elements, or more constrained visual settings. However, systematic evaluation of understanding over realistic, large-format engineering drawings remains limited. To address this gap, we introduce LED-BENCH, a multidisciplinary benchmark designed to evaluate fine-grained understanding and complex reasoning over large-format engineering drawings, spanning the mechanical, architectural, and transportation domains. Extensive evaluation of 11 representative multimodal large language models, including 8 proprietary models and 3 open-source models, reveals substantial remaining difficulty: GPT-5.6 Sol achieves 52.65% average accuracy, performance varies across engineering domains, and all evaluated models perform worse on questions requiring broader evidence integration. These results highlight fine-grained understanding, evidence integration, and complex reasoning over large-format engineering drawings as persistent challenges for current multimodal models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.