Training Small Language Models for Transferable Tool Selection
Tool selection across corpora of tens of thousands of APIs is a critical bottleneck for LLM agents, because standard retrievers rank by relevance without knowing which tools a multi-step instruction actually requires. This paper post-trains small open Qwen3 models (1.7B to 8B parameters) via supervised fine-tuning and reinforcement learning to decompose a request and select tools through a lightweight Plan–Retrieve–Select workflow. The headline result is that an 8B checkpoint outperforms a 402B-parameter Llama-4-Maverick on ToolBench and transfers zero-shot to unseen tool domains and retrievers, suggesting that tool selection can be compressed into a small, locally hostable policy rather than delegated to massive proprietary models.
The paper delivers a clean, well-scoped contribution: a reproducible training recipe that lifts small models past a much larger prompted baseline on an exact-match tool-selection task. The methodology is clearly described and the transfer experiments across corpora and retrievers are genuinely useful, though the claim of dominating Llama-4-Maverick is most pronounced when the large model is confined to the authors' own workflow rather than its best-performing alternative.
The decomposed workflow itself is the strongest element: separating planning from selection keeps per-call context bounded and makes the policy retriever-agnostic. The demonstrations of zero-shot domain transfer (Gorilla, CRAFT, TaskBench) and retriever transfer (BM25 to COLT) support the central claim that the learned behavior is generic instruction decomposition and description matching rather than corpus-specific memorization.
The workflow rests on a strong structural assumption that limits applicability: each substep must map to exactly one independent tool with no inter-step dependencies. The authors acknowledge this but do not quantify how often real tasks violate it.
On TaskBench, the small model actually underperforms a flat retrieval-and-reasoning style using the same large baseline, revealing that decomposition can hurt when domain knowledge is needed. The exact-match metric is unforgiving, and the reported single-run results exhibit minor non-determinism that leaves uncertainty about variance.
The in-domain evidence is solid—the 8B Qwen3 reaches 41.5% exact-match success versus 30.6% for the best Maverick prompting setup on ToolBench G3—but the comparison carries important caveats. The SFT data are generated entirely by Llama-4-Maverick, meaning the small model is trained to imitate the same teacher that also filters gold labels and generates planning targets, blurring the line between distillation and independent capability. The evaluation metric penalizes any deviation from the filtered gold set, yet the teacher model is used to decide which gold tools are truly necessary, which embeds its judgment into the ground truth itself.
While the authors document prompts, hyperparameters, and software versions thoroughly, they do not release code, checkpoints, or the curated SFT data, which blocks independent reproduction. Training requires full fine-tuning on 8×A100 GPUs, so the barrier is nontrivial even with the reported details. Additionally, evaluation numbers come from a single run, and the authors note visible run-to-run variation caused by non-deterministic batched inference.
As LLM agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step user instruction becomes a central bottleneck. Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed. Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever, decomposing the instruction and selecting over retrieved candidates. We show that a small, open model can be trained via reinforcement learning to perform the same functionality. On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B). In addition, the trained model transfers directly to different retrievers and generalizes to new tool corpora in different domains.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.