NavAble: A Large-Scale Dataset and Synthetic Data Generation Pipeline for Blind Navigation

The synthetic data generation pipeline is the paper's strongest technical contribution. By chaining automated web crawling, Gemini-based filtering, Grounding DINO localization, SAM2.1 segmentation, and SAM3D reconstruction, the authors convert uncurated web images into 500 simulation-ready 3D assets and render 452K annotated frames with paired depth, normals, and bounding boxes. The per-class analysis is convincing: modern architectures make disproportionate gains on thin structural classes such as pedestrian signals and elevator buttons, and the evaluation on two distinct assistive robot platforms shows that improvements are not purely benchmark artifacts. The paper also deserves credit for releasing the dataset, asset library, Isaac Sim extension, and training details.

The paper contains a significant factual discrepancy in its central claim: the abstract promises >80% $\text{mIoU}$ on the held-out test set, but the best fine-tuned result reported in Table 1 is 72.74% $\text{mIoU}$ (Mask2Former), with the next best at 68.12%. This overstatement undermines confidence in how the gains are communicated. Beyond inflated headlines, the experimental story is uneven across architectures. While transformer-based and mask-classification models benefit from synthetic data, CNN per-pixel models such as DeepLabv3+ and EncNet degrade by up to 10 points when synthetic data is added, indicating that the sim-to-real gap is not fully bridged and synthetic augmentation is not universally safe. Most critically, absolute performance on embodied out-of-distribution platforms remains poor—SegFormer achieves only 16.17% $\text{mIoU}$ on the cane-shaped robot and 23.08% on the quadruped—suggesting that benchmark $\text{mIoU}$s in the 60s and 70s do not yet translate to reliable real-world navigation perception.

The evidence broadly supports the claim that accessibility-critical classes are underrepresented in standard datasets, and the comparisons to prior BLV datasets are fair. The authors correctly note that Islam et al. provide image-level presence labels rather than pixel-level segmentation masks, and that obstacle-detection datasets like SurDis and GuideTWSI cover narrower scopes. However, the comparisons to zero-shot VLMs are used only as indicative reference points rather than reproducible baselines, which the paper acknowledges but does not rigorously benchmark against. The synthetic augmentation gains are supported by consistent per-class trends across multiple architectures, and the location-level train/test split is a strong design choice. Still, the absence of error bars or statistical significance tests—explicitly acknowledged in the checklist as a limitation—makes it impossible to judge whether reported $\text{mIoU}$ deltas are stable.

The authors release the full dataset, Isaac Sim extension, and training code, which satisfies the checklist requirements and represents good community practice. Appendix A provides detailed per-architecture hyperparameters, optimizers, learning-rate schedules, and iteration budgets. Nevertheless, full reproduction faces practical barriers: the synthetic pipeline depends on the proprietary Fab marketplace for environments and on a proprietary VLM (Gemini) for image filtering, introducing version and licensing drift. The checklist also admits that no error bars or repeated seeds are provided, so independent researchers cannot easily assess variance. Finally, the paper notes that portions of the synthetic pipeline still require human intervention for object placement and trajectory recording, meaning the generation process is not yet fully automated.