Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
Typed decision models route requests or select tools from predefined categorical options, using fallbacks such as "Other" or "Defer" to handle out-of-scope inputs. This paper introduces the Paired Fallback Robustness (PFR) Test to check whether models correctly switch to fallback when required and whether they remain consistent under logically equivalent rule descriptions such as Union versus Complement definitions. The authors find that fallback mechanisms frequently fail silently: Intern's accuracy on TREC50 collapses from 97.9% to 23.1% on correctly classified inputs when reassigned to fallback, and equivalent definitions can flip individual decisions without changing aggregate accuracy. These results matter because fallback options, intended as reliability safeguards in production routing and delegation systems, can themselves become a source of error.
The paper makes a compelling case that fallback reliability in typed decision models is substantially weaker than aggregate accuracy suggests. Across three benchmarks and multiple architectures, the four-request pass rate $P_4$ ranges from $18.2\%$ to $81.0\%$, with Intern/TREC accuracy collapsing from $97.9\%$ to $23.1\%$ on correctly classified inputs when reassigned to fallback. The authors conclude that "fallback mechanisms designed to improve reliability can instead introduce errors," a finding that upends the common assumption that adding a fallback catch-all is sufficient for safe deployment.
The PFR framework is elegant and diagnostically powerful. By crossing two assignments with two equivalent definitions, it jointly tests whether models respond to changed rules and ignore surface wording changes. The category-information diagnostic cleanly separates recognition failures from assignment failures, showing that "correct classification in the reference task does not guarantee correct fallback selection."\n\nThe empirical breadth across four model families, three datasets, and controlled variations in position, wording, and policy formulation gives the failures strong external validity within the studied class of TDMs.
The study identifies failures but does not propose or evaluate any mitigation, explicitly limiting its practical utility. As noted in the Limitations, "This work focuses on identifying and systematically evaluating fallback reliability failures in TDMs rather than developing mitigation methods."\n\nAdditionally, the Defer-policy extension in Section 4.3 reports that "Joint policy correctness is zero for all four models under both criteria," but it rests on only 128 hypothetical texts per panel—far smaller than the main benchmarks. Some diagnostics also use test-set labels to define oracles, making them insightful ablations rather than realistic deployable pipelines.
The evidence broadly supports the central claims, with Table 1 and Appendix figures showing that U/C disagreement is pervasive even when net accuracy differences are small. The authors fairly situate their work against adversarial TDM studies and concurrent work on rejection behavior, noting that prior literature has not examined non-adversarial fallback execution.\n\nOne caveat is that several cited papers (e.g., Hu et al., 2026; Sun et al., 2026) are dated 2026 and therefore unverifiable at this time, which slightly weakens the related-work anchoring. Within the paper, however, the paired-sample statistics and Holm-corrected McNemar tests are appropriate, and the matched comparisons across definitions are well-controlled.
Reproduction is hampered by the authors' decision to withhold code and data until acceptance: "We will make all code and data publicly available upon acceptance of the paper." While model interfaces and sampling procedures are documented, some hyperparameters—such as Kev’s temperature of 2.40605—appear oddly precise without rationale, and key ablations are buried in appendices.\n\nThe fixed-score threshold analysis is derivable from the equations given, but independent reproduction would require exact prompt templates, category descriptions, and hash-based sampling ranks that are only partially specified. Without released artifacts, an independent team would struggle to replicate the exact paired panels.
Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.