TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
Most multimodal benchmarks test perception or reasoning over determinate inputs, leaving cross-modal ambiguity resolution largely unmeasured. TRIAD introduces a diagnostic benchmark of 327 text-image-audio triplets formally constructed so that only the full triplet yields a unique answer, while any two modalities leave multiple options viable. By organizing items into 86 scene groups and evaluating 14 omni-modal systems under closed and open-ended conditions, the paper shows that strong query-level accuracy masks fragile scene consistency and that removing answer options collapses models to single-digit group accuracy, reframing evaluation around tri-modal disambiguation rather than recognition.
TRIAD is a conceptually strong diagnostic benchmark that isolates tri-modal mutual disambiguation through a well-defined formal criterion and a rigorous group-level metric. The headline finding—that the best model reaches 71.2% group accuracy while humans score 91.6%—is compelling evidence of a real failure mode. However, its impact is constrained by very small scale (327 queries), a narrow human baseline on a 100-query subset, and reliance on proprietary judge models for open-ended grading. High leave-one-out scores furthermore challenge the construction's irreducibility claim. It is a valuable proof-of-concept and error-analysis tool, but not yet a definitive leaderboard for omni-modal reasoning.
The formalization of well-posed tri-modal disambiguation gives the benchmark a clear theoretical foundation, and the choice of group accuracy $\mathrm{Acc}_{\mathrm{g}}$ as the primary metric directly penalizes unstable scene interpretations in a way that isolated query accuracy cannot. The tiered ambiguity analysis and comprehensive leave-one-out ablations are well-executed and produce interpretable diagnostics. As the authors note, the sharp collapse from multiple-choice to open-ended answering reveals how deeply current systems depend on explicit option scaffolding rather than generating a resolved interpretation.
At only 327 queries and 86 scene groups, TRIAD is too small to draw firm conclusions about 14 different model families, and the human baseline is measured on a cost-controlled 100-query subset that may not be representative. More troubling is the empirical violation of the irreducibility hypothesis: under leave-one-out ablation, Gemini 3.1 Pro still achieves 57.2% group accuracy without the image and 51.8% without the audio, scores the authors attribute to shortcut dependence but which also raise the possibility that manual well-posedness checks failed to catch subtle textual or prior leaks. Additionally, grading open-ended responses with judge models is circular and under-validated; inter-judge agreement is modest ($\kappa \approx 0.55$ in Table 10) and human adjudication is used only on disagreement items, introducing uncertainty into the headline open-ended collapse.
The paper fairly positions itself against prior work. The retrieved OmniBench abstract confirms that it targets integrated tri-modal reasoning without enforcing the mutual-disambiguation or group-consistency properties that TRIAD introduces. The internal evidence largely supports the main claims: the ambiguity-tier breakdown shows monotonic degradation as more modalities carry internal ambiguity, and the group-size collapse demonstrates that models fail on consistency rather than isolated difficulty. These controls strengthen the argument that the gap is diagnostic of disambiguation failure rather than generic task hardness.
The paper documents core hyperparameters ($\text{temperature}=0$, one sample per query) and promises release of a reference well-posedness checker and source metadata, which aids reproducibility. Unfortunately, no public repository URL, dataset download link, or exact API snapshot date appears in the provided text, and the reliance on future proprietary judge models for open-ended evaluation means those scores may shift as model versions update. The manual construction pipeline—five annotators, ~40% admission rate, and human-only well-posedness gates—is labor-intensive and not easily scalable without the released tools.
Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.