Prompt Sensitivity in Tool-Augmented Model Comparisons
Language model rankings on question-answering benchmarks are known to shift with prompt wording, raising doubts about single-prompt evaluation. This paper investigates whether augmenting models with deterministic tools stabilizes pairwise comparisons, proposing a matched protocol that isolates answer readout, reasoning, tool invocation, and report access to attribute changes in wording sensitivity. Applying the protocol to six open models across five domains and five prompt wordings, the authors find that tools improve accuracy where a program directly answers the question yet frequently widen ranking instability because call frequency itself becomes wording-dependent. The work matters because it shows that deterministic tool output is not enough to guarantee stable evaluation, and it recommends reporting call frequencies and pairwise stability alongside headline accuracy.
The paper convincingly demonstrates that tool augmentation does not inherently stabilize model comparisons and can amplify wording sensitivity when call behavior depends on prompt phrasing. The matched protocol is well-designed and the call-frequency decomposition usefully attributes variation, though the results are bounded by a small set of wordings, models, and domains and the statistical tests are explicitly exploratory.
The matched diagnostic protocol is the paper's strongest asset: by holding items constant while varying readout position, reasoning, tool access, and interface type, the authors cleanly separate sources of instability. The identity $\mathrm{acc}(w)=a^{0}_{w}+c_{w}\,(a^{1}_{w}-a^{0}_{w})$ and its Shapley decomposition into call-frequency and conditional-accuracy components provide an elegant attribution of wording effects without claiming causality. The inclusion of seven call-level interventions further strengthens the mechanistic interpretation.
The study's generalizability is limited: only five domains and five English wordings are tested, the spread statistic grows with more wordings, and the authors caution that 'five wordings support little inference to new ones.' The statistical contrasts are explicitly exploratory because the bootstrap and exchange tests are not proven to hold the nominal Type I error rate when conditions differ in accuracy level, making the Holm-corrected counts suggestive rather than definitive. Additionally, three models run at 8-bit precision, which may alter generation behavior and call frequencies compared to native-precision inference.
Within its scope, the evidence strongly supports the central claim that tool loops widen pairwise spreads when call frequencies vary with wording: the plain-text loop widens 37 of 60 molecule pairs against reasoning, and native-interface ablations reduce the widening by stabilizing calls. Controls such as constant tools and supplied reports effectively isolate information access from invocation behavior, and the results are broadly consistent with prior work on prompt sensitivity in readout and paraphrase. The related work discussion is adequate but does not engage deeply with why some interfaces eliminate sensitivity while others do not.
Reproduction is partially supported by thorough documentation: models, revisions, token budgets, tool descriptions, and exact wordings are detailed in supplementary material, and the authors state that 'every answer, call line and trajectory analysed here was generated by these models under the conditions described, with greedy decoding, on vLLM.' However, no code repository, data release, or analysis scripts are mentioned in the main text, which would block exact reproduction. The use of 8-bit quantization for three models and the absence of a full-size repeat for mathematics further complicate independent replication.
Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.