Separate Field Scoring Protects Unused Tools When Agents Pool Tool-Use Experience
When agents share a tool catalog and pool their usage experience, adding that usage to the retrieval index boosts tools with usage but can push unused tools out of the top results. This paper shows that folding usage examples into the index severely lowers held-out Hit@5 on two of three catalogs, and proposes Synapse, which scores documentation and usage as separate z-normalized fields before combining them with a learned weight. The finding matters because multi-agent systems risk silently failing on rarely-used tools unless the combination rule explicitly protects them.
The paper makes a targeted, well-controlled contribution: it isolates a real retrieval failure mode for held-out tools and proposes a simple fix that dominates a slot-reserving router on held-out queries across all tested catalogs and embedders. That said, the practical importance of the finding is catalog-dependent—nearly vanishing on ToolRet—and the mechanism behind Synapse's held-out protection is not fully disentangled from a mere binary has-usage signal.
The experimental controls are the strongest part. The length-matched text control and the single-entry budget control convincingly show that the held-out damage comes from structural index competition or entry length, not from usage semantics, while the labeled gain genuinely requires correct content. The careful split construction—salted SHA-256 hashes, mention filtering, and bucket-independent assignments—makes the held-out/labeled distinction credible.
Three limitations weaken the broader claims. First, the effect is highly catalog-dependent: on ToolRet, folding barely moves held-out Hit@5 (0.306 to 0.290), and Synapse does not improve it over documentation-only, yet the paper offers no explanation for why the damage is so small there. Second, the mechanism behind Synapse's held-out protection is not fully isolated: a constant offset added to the documentation score of tools with usage matches Synapse on held-out queries in five of six settings, which suggests the separate field may be acting largely as a calibrated has-usage flag rather than leveraging usage content for held-out protection. Third, the federated pooling experiment (RQ3) is only run on StableToolBench, shows no benefit beyond centralizing the same records, and uses simulated agents rather than a real distributed deployment.
The evidence supports the core claim that folding hurts held-out tool retrieval on two of three catalogs, with both embedders. However, comparisons to related work are thin because the authors explicitly did not test learned retrievers such as Tool-Embed or Re-Invoke, nor did they measure end-to-end task success. The trade-off between Synapse and the router is clearly documented—Synapse wins on held-out, router wins on labeled for TOUCAN and ToolRet with Jina—but the paper's recommendation that the choice depends on the workload is undercut by having computed the break-even share only on StableToolBench.
The paper is unusually detailed about its experimental protocol: frozen weights and quotas, exact tokenizer-level length matching, salted hashes for held-out splits, and paired bootstrap intervals clustered by tool are all reported. Hyperparameters are explicitly listed ($w = 0.5$, $d = 4$ on StableToolBench; $w = 0.9$, $d = 4$ on ToolRet; $w = 0.9$, $d = 3$ on TOUCAN). The main barrier to independent reproduction is that no code repository or data-preprocessing scripts are mentioned, and the exact serialization format for documentation versus usage entries is not provided. Additionally, the incomplete audit of Nomic's training data leaves a contamination question unresolved.
Agents that call tools from a shared catalog can pool what they learn about which tool serves which request. We measure what pooled usage examples do to tool retrieval, reporting separately the held-out tools, for which no agent has usage, and all other tools. Adding usage examples to the index as extra entries for used tools raises Hit@5 (whether a gold tool is in the top five) for those tools and lowers it for held-out tools, by a large margin on two of three catalogs and with both embedders we tested. Giving used tools the same number of equally long entries filled with other tools' text lowers held-out Hit@5 about as much, but gives little or none of the gain. We call our method of scoring usage as its own field SYNAPSE. On StableToolBench, the held-out gold tool appears in its top five more often than with a router that reserves slots for usage (0.785 against 0.683), and this held-out lead holds on all three catalogs with both embedders. This protection persisted when we replaced each tool's usage with other tools' text, and a constant score offset for tools with usage gives similar held-out Hit@5 in most settings. The gain for tools with usage, in contrast, needs usage about the right tool. The router keeps more of the gain on labeled queries on TOUCAN with both embedders and on ToolRet with one, so the better choice depends on the workload. Splitting the usage among ten simulated agents shows that pooling helps exactly as much as giving one agent the same records.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.