Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Document understanding has been evaluated through isolated silos—layout detection, OCR, reading order, and image captioning—making it impossible to audit whether a single coherent page parse exists. This paper proposes Read-Parse-Describe, a unified benchmark built from 2,402 ACM scientific documents that forces models to jointly output bounding boxes, semantic labels, reading order, recognized text, caption links, and alternative-text descriptions for visual elements. To combat spatial drift in compressed vision-language tokens, the authors introduce SARA, a lightweight residual adapter that re-injects crop-aligned multi-layer visual features into frozen VLMs.
The paper makes a valuable conceptual contribution by recasting document parsing as a spatially grounded binding problem where layout, text, reading order, and alt-text must agree inside a single structured element record. The Read-Parse-Describe dataset and SARA adapter are well-motivated, and the experiments show that frozen-backbone adapter tuning can substantially improve structural metrics. However, the benchmark is narrowly scoped to the first eight pages of ACM papers, the alt-text ground truth relies on an LLM-in-the-loop annotation pipeline whose reliability is only spot-audited, and SARA's gains over full fine-tuning only emerge with at least half of the training data. The inclusion of numerous proprietary and unreleased baselines also weakens the comparative claims.
The unified output schema is the strongest idea: by requiring bbox, category, order, text, caption_id, and alt_text to live in one element object, the benchmark turns orphaned captions and misaligned descriptions into measurable failures rather than silent omissions. SARA is a clean architectural fix for spatial compression in VLMs; the ablation shows that crop-aware alignment combined with multi-layer feature reuse yields large F1 and mIoU gains, while projector-only training actually underperforms full fine-tuning, confirming that the gains are structural and not merely from extra parameters.
The dataset is highly constrained to open-access ACM PDFs and primarily the first eight pages, leaving references, appendices, and non-academic domains almost entirely out. Alt-text annotations are bootstrapped with Gemini 3 Pro, scored by Gemini 3 Flash, and only human-refined when the score falls below 4; while the authors disclose this honestly, a 200-sample audit cannot fully validate 11,338 descriptions. Worse, the evaluation of alt-text uses METEOR and BERTScore, which measure lexical overlap rather than whether a description actually conveys the necessary visual information to a blind reader. Baseline comparisons also lean heavily on proprietary API-only models and unreleased checkpoints, making independent verification impossible.
The evidence supports the claim that existing pipelines are fragmented—MinerU reaches 0.9671 OCR accuracy but 0.6757 reading-order score and produces no alt-text under the schema—yet the comparison is uneven because many baselines are API-only or unreleased systems. SARA's gains are solid on Qwen3.5-2B but inconsistent across backbones: on Gemma4-E4B it improves layout F1 while decreasing reading-order and description scores, suggesting the adapter is not universally beneficial. The paper does report this trade-off, which is fair, though the framing sometimes emphasizes layout wins over the more mixed picture.
Reproduction is hampered by missing training details and dependence on closed infrastructure. Hyperparameters for SARA (learning rate, batch size, schedule) are entirely omitted; only trainable parameter counts are reported. The evaluation protocol relies on unreleased baselines and proprietary APIs (ChatGPT, Gemini), whose exact prompts, schema conversion scripts, and API versions are not provided. While the authors promise dataset and code release on GitHub, no commit hash, DOI, or permanent archive is offered at submission. Reconstructing the alt-text labels also requires paid access to Gemini 3 Pro and Flash, creating cost and version-lock barriers.
Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.