From Relevance to Validity: Zero-Shot Tool Selection via Functional and Contextual Validation
Zero-shot tool selection for LLM agents is brittle under distribution shift because semantic relevance alone does not guarantee that a tool's endpoints support the required operation or remain applicable in a new context. This paper proposes TACT, a progressive validation framework that augments semantic retrieval with endpoint-grounded functional compatibility and task-conditioned contextual evidence. The core claim is that explicitly separating relevance, functional compatibility, and contextual applicability closes the transfer-validity gap and yields stronger ranking under cross-domain and cross-dataset transfer.
TACT decomposes zero-shot capability transfer into semantic relevance, endpoint-grounded functional compatibility, and task-conditioned contextual applicability, and reports strong aggregate ranking gains over strong baselines on controlled candidate views. The staged design is well motivated by preliminary diagnostics, yet the evaluation relies on synthetic hard-negative views and a small adjudicated minimal-pair set, while the absence of released artifacts limits external validation. The validated ranking score $s(t,c)=A(t,c)+\omega_I(t,c)[I(t,c)-A(t,c)]+\omega_D(t,c)D(t,c)$ uses bounded gates that retain the preceding estimate when a correction lacks evidence, and the paper reports that "On the cross-domain Context view, TACT improves NDCG@5 from 0.518 for the strongest baseline, PORTS, to 0.624," while stressing that "All model strengths, checkpoints, and fusion parameters are selected using source contexts only."
The controlled diagnostic in Figure 2 is compelling: among eligible tasks, "44.25% contain functionally different candidates with semantic scores less than 0.01 apart," showing that semantic near-ties hide functional incompatibilities. The progressive architecture—Semantic Proposal, Functional Validation, Contextual Validation, and Validated Ranking—is cleanly derived from this observation, and the ablation confirms that removing functional validation hurts cross-domain NDCG@5 by substantial margins, e.g., "0.067 on Context." The use of bounded gates in $s(t,c)=A(t,c)+\omega_I(t,c)[I(t,c)-A(t,c)]+\omega_D(t,c)D(t,c)$ to suppress unsupported corrections is a principled safeguard against hallucinated adjustments.
The evaluation heavily depends on hand-constructed candidate views (Semantic, Context, Mixed) that inject hard negatives into a closed-world pool, so the metrics partly measure recovery of adversarially inserted distractors rather than open-world validity. The 15.20% contextual-flip rate motivating Contextual Validation derives from a small minimal-pair set judged via "automated assessment with author adjudication" on only 408 determinate pairs, which is suggestive but fragile.
Moreover, the requirement extractor and endpoint-schema matching assume clean, public API documentation—a condition that breaks for poorly documented or proprietary endpoints. The paper itself acknowledges these limits: "The reported ranking gains therefore do not establish performance for newly added capabilities, changing endpoint schemas, or live services," and adds that "the closed-world relevance annotations treat unjudged candidates as non-relevant in expanded pools, so the scaling results measure recovery of known positives rather than complete compatibility assessment."
Baseline comparisons are conducted under a shared ranking interface, which is fair in principle, but methods like ToolDreamer and Re-Invoke are run in their "original configurations" without retraining on the shared source data, while TACT is optimized end-to-end for this protocol. The downstream execution experiment on 120 cached StableToolBench queries shows consistent completion gains across four frozen backbones, yet absolute rates remain low, and the paper cautions that "the cached execution study uses a fixed set of solvable queries and frozen agent backbones; its completion rates may not transfer to unrestricted tasks, live API failures, or other agents," tempering the claim that ranking improvements reliably translate to real-world task completion.
No code, data, or checkpoints are released, and the methodology depends on numerous implementation details buried in the appendices—including exact partial injective assignment over endpoint sets and cross-fitted source-profile construction—that are difficult to replicate without the original pipeline. A further subtlety is transductive coordinate alignment: as the paper notes, "Before cross-dataset scoring, task and capability embeddings are statistically aligned using each dataset's own unlabeled means and standard deviations. This includes the held-out target's unlabeled statistics and is therefore transductive preprocessing." Although target labels are excluded, this alignment requires access to the full target embedding distribution at test time, complicating independent reproduction on unseen datasets. The paper also discloses that "A deterministic task-only extractor splits a task into action clauses, attaches local constraints, merges near-duplicates, and retains at most $N_{\\max}=4$ requirements," yet how this extractor is implemented is not specified.
Large language model (LLM) agents increasingly face tasks from unseen domains and execution contexts, making zero-shot tool selection essential for identifying tools that can satisfy task requirements. Existing dominant methods mainly rely on relevance ranking, but semantic matching alone cannot ensure that a tool is functionally compatible or remains applicable in a new context. We preliminarily identify a \emph{transfer-validity gap}, where semantically relevant tools may still fail to satisfy functional or contextual requirements after transfer. To address this gap, we propose \methodname{}, a progressive validation framework that complements semantic relevance with endpoint-grounded functional compatibility and task-conditioned contextual evidence. Across cross-domain and cross-dataset transfer, \methodname{} consistently improves capability ranking, with larger gains under functional or contextual ambiguity. Ablation results further confirm the complementary contributions of functional and contextual validation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.