Typed Decision Models on Tabular Data
Typed decision models return probabilities over fixed answer choices instead of generating free text, making them a natural fit for tabular prediction tasks. This paper evaluates four such models against three large language models on row classification, fact verification, and anomaly detection, subjecting every request to 21 content-preserving rewrites. The authors find that decision models match LLM accuracy at far lower cost and are far more robust to how questions are phrased, yet no model escapes fragility when the table layout is altered.
The paper is a solid, timely empirical study that establishes decision models as a practical alternative to LLMs for fixed-answer tabular tasks. Its systematic rewrite framework and focus on per-decision flips rather than aggregate accuracy alone are real strengths. However, the central negative result—that every model remains vulnerable to table formatting—limits the practical reassurance it offers, and the reliance on single snapshots of closed APIs weakens generalizability.
The experimental design is rigorous and comprehensive. Sending every request in its original form and under 21 rewrites creates a clean, paired comparison of robustness that was missing from prior work. The inclusion of Clef and Kev-27B, both trained from Qwen3.8-27B, isolates the effect of the decision-model objective from backbone differences. The link between returned probability and fragility, measured via flip AUROC and re-querying strategies, is methodologically convincing.
The headline promise of decision models is undercut by the finding that none are robust to how the table is organized, with roughly a third of row decisions flipping under some format or order rewrite. It is also concerning that GPT-6 Luna Decisions—built on an LLM and served through a Decisions API—mirrors LLM fragility to option ordering, showing that the decision interface alone does not confer robustness. The study is further limited by testing each model in only one version, leaving open whether these patterns hold across updates or training runs.
The main claims are well-supported by the evidence. Table 3 shows parity in accuracy with zero generated tokens for decision models, while Figure 3 and Table 5 quantify flip rates across rewrites. The comparison to related work is adequate but mostly enumerates known LLM vulnerabilities rather than explaining why decision models trained with proper scoring rules or RLCD should behave differently. The authors fairly acknowledge that no prior arXiv study had evaluated decision models as a method for tabular data.
Reproducibility is partial. The open-weight models use exact commits and documented hardware, and decoding hyperparameters are reported. However, the two commercially closed models are black-box APIs whose internals are hidden and may change, a limitation the authors explicitly flag. No public code or evaluation scripts are mentioned, so reproducing the rewrite suite and metrics would require rebuilding the pipeline from the paper. The benchmarks themselves are standard (TabLLM, TabFact, ReTabAD), which aids dataset-level replication.
Typed decision models are language models that answer a question with a fixed set of possible answers by returning a probability for each answer instead of generating text. We study them on tabular data and compare four decision models with three large language models (LLMs) on row classification, fact verification and anomaly detection. Every request is sent in its original form and under 21 types of rewrite that keep its content, such as renaming the options or writing the table in another format. The decision models perform as well as the LLMs at a lower cost. Three of them are also far more robust to how the question is formulated. No model we test is robust to how the table is organized. For every model, at least a third of the row decisions change under some rewrite of the table or the question. The decisions to which a decision model gives low probability are the ones most likely to change. Decision models are a practical choice for tabular decisions, but their robustness remains an open problem.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.