VLTest: Finding Joint Image-Text Failures in Vision-Language Models
VLTest addresses the problem of testing vision-language models (VLMs) where mutating an image, a text prompt, or both can change the expected answer, making naive output-change counting unreliable. Its core idea is to generate controlled visual variants via VQGAN latent code editing and textual variants via a rewrite–expand–prune pipeline, then confirm failures with an automated oracle and attribute joint failures to the specific image-text combination using matched four-cell evidence. This matters because VLMs are increasingly embedded in software systems, and existing robustness evaluations systematically overcount failures by ignoring whether an input still has the same correct answer.
VLTest is a conceptually clean and empirically thorough black-box testing framework that fills a real gap in multimodal software testing. The paper rigorously defines confirmed failures and attributed joint failures via an automated oracle, and the four-cell design provides stronger causal evidence than prior work. That said, the approach depends heavily on the accuracy of a VLM judge (Llama-4-Maverick), and some generalization claims rest on small-scale API experiments with only 50 seeds and a budget of 50 calls, which limits confidence in cross-model portability. As the authors note, "The oracle relies on a VLM judge, which can make mistakes," and while precision against human labels reaches 88.8% to 94.6%, unmeasured false negatives could propagate through the entire pipeline.
The strongest aspect of the paper is its crisp articulation and empirical validation of why output compatibility alone is insufficient for VLM testing: across all input types, 81.7% of candidates are confirmed failures, while 14.8% reflect changed expected answers the filters miss. The custom generators are effective and score-agnostic, with VLTest-V and VLTest-T achieving 1.8 to 2.5 times the confirmed-failure yield of SINVAD-VQ and TextFooler. The stable-first pairing is an elegant efficiency win: as shown in the results, "Stable-first ordering therefore does not make a stable pair more likely to fail; it spends the joint budget only on pairs that can yield an attributed joint failure, at no additional cost."
The most consequential weakness is circular reliance on a black-box VLM judge to validate inputs for another black-box VLM under test. The paper concedes that "the judge receives the reference answers as a reference, which may bias its answer check toward preserved," yet every attribution chain and the headline precision metrics are downstream of this unverified bias. In addition, the generalization experiment on commercial APIs (GPT-4o mini and Gemini 2.5 Flash) uses only 50 seeds and a joint budget of 25 calls, so the reported 112–250 attributed joint failures per 1,000 calls are extrapolated from tiny samples and lack the paired statistical testing applied to the primary experiment. The margin analysis for interaction failures is also fragile: for PaliGemma, 622 failures (3.2%) had to be excluded because the additive score $\eta(x,t)$ under teacher forcing contradicted the decoded output, suggesting the generative setting may break the score-based contrast model.
The paper's evidence supports its core claim that four-cell confirmation and relation filtering improve failure attribution and yield, though headline comparison ratios should be read carefully. The 2.8× improvement over MetaRA-c uses a full-budget metric that MetaRA-c cannot utilize because it exhausts its fixed relation combinations after about half the budget; when normalized by actual calls, MetaRA-c achieves roughly 128 attributed joint failures per 1,000 joint calls versus about 185 for VLTest (SF), a narrower but still meaningful gap. The baseline comparison against SINVAD-VQ and TextFooler is fair and well-controlled under identical budgets and accounting rules, and the authors correctly exclude adversarial attacks from comparison because those methods "assume rather than check that the expected answer is preserved."
The authors pledge a replication artifact and document most implementation details, including VQGAN hyperparameters ($\kappa=15$, $R=8{,}191$, 256$\times$256 resolution), text-pipeline constants ($\gamma=\tau=\delta=5$, $\beta=4{,}000$), and judge prompts fixed on an excluded 20-seed pilot. However, reproducing the commercial API experiments is brittle because "The API models are called at temperature 0 where the API allows it, and every call is logged with its date," implying snapshot-dependent and non-deterministic behavior that is not under the authors' control. Missing public release of the exact judge prompts and the partial dependence on an unreleased artifact also create practical friction for independent teams.
Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.