Multi-Tool Selection for Robotic Systems: Comparing Semantic Retrieval and LLM
Natural-language robot control requires mapping user requests to predefined tool subsets without inventing new capabilities. This paper frames tool routing as bounded multi-label selection over a fixed catalog and compares three ChromaDB retrieval policies against a cloud-hosted 120-billion-parameter LLM. The core finding is a clean quality-latency trade-off: clause-aware retrieval (CDBv3) runs locally in milliseconds but lags the LLM in routing accuracy, making it suitable for connectivity-constrained or privacy-sensitive deployments.
The paper presents a focused and well-motivated empirical comparison, but its conclusions are weakened by methodological shortcuts. The authors are commendably transparent about tuning CDBv1–v3 hyperparameters directly on the 164-prompt evaluation set, which renders the retrieval scores optimistic upper bounds rather than unbiased estimates. Additionally, the LLM baseline is identified only as 'gpt-oss:120b-cloud(digest 569662207105)' with no citation or provider documentation, raising concerns about reproducibility and whether the comparison is against a verifiable model.
The problem formulation is crisp: routing is treated as $f: \mathcal{X} \rightarrow 2^{\mathcal{T}}$ with exact-match accuracy, completeness, and precision, which cleanly separates selection from downstream planning. The progressive refinement from CDBv1 to CDBv3 is logical, and the authors avoid overstating retrieval superiority. They explicitly note that 'The results therefore do not establish retrieval as a universally more accurate replacement for large-model routing,' and frame CDBv3 as a practical local alternative when latency or connectivity are constraints.
First and most seriously, all retrieval hyperparameters were tuned on the final test set of only 164 prompts, meaning the reported CDBv1–CDBv3 scores likely overstate true generalization performance. The authors admit this but do not correct for it. Second, the benchmark is tiny and hand-annotated by the same domain expert who designed the 42-tool catalog, creating a risk of confirmation bias. Third, a promising edge baseline—a small local LLM (7–8B)—was dropped from the comparison because its results were 'discouraging,' leaving open whether a stronger compressed model could close the gap. Finally, the LLM baseline is opaque: 'gpt-oss:120b-cloud' is not a standard, citable model, so the claimed quality advantage is difficult to verify.
The evidence supports the high-level claim that local clause-aware retrieval trades accuracy for speed, with CDBv3 scoring $\mathcal{L}=0.7822$ against the LLM's $0.8869$ and running in milliseconds versus seconds. However, the small sample size and absence of confidence intervals or significance tests make it impossible to judge whether the observed gaps are robust. Comparisons to related work are fair: the authors clearly distinguish their routing boundary from prior task-planning work such as ReAct and Code as Policies, and they do not claim their retrieval methods outperform general tool-use agents on broader metrics.
The paper does not mention a code or data release, and the evaluation dataset of 164 prompts is not publicly linked. The CDB hyperparameters—fixed threshold $\tau_{\mathrm{abs}}=0.85$, relative margin $\rho=0.60$, and aggregation gap $0.28$—were tuned by hand on the final benchmark, so independent reproduction would need a fresh development split to avoid leaking test information. Hardware is specified down to an NVIDIA RTX 5060 Ti, but the cloud model 'gpt-oss:120b-cloud' lacks a citation or API documentation, making the LLM baseline impossible to verify. Additionally, no random seed, environment file, or embedding checkpoint is provided.
Natural-language interfaces allow users to command robotic systems without interacting directly with mission planners or robotics middleware. Their reliable use depends on robot tool routing, defined here as the selection of the smallest complete subset of predefined executable capabilities required to satisfy a request. This task is particularly difficult for compound instructions, where incomplete selection may omit necessary sensing or monitoring capabilities, while excessive selection may expose irrelevant tools. We compare four routing strategies: three progressively refined ChromaDB-based methods - fixed-threshold retrieval, relative-distance retrieval, and clause-aware aggregated retrieval - and a cloud-hosted 120-billion-parameter language-model baseline. The methods are evaluated on 164 manually annotated prompts organised into three complexity levels, from direct single-tool requests to compound requests requiring at least three tools. Using exact-match accuracy, completeness, precision, a completeness-weighted composite score, and routing latency, CDB-v3 achieves the best macro-average score among the retrieval-only methods, with 0.7822, compared with 0.8869 for the large-model baseline. Although the LLM provides higher routing quality, CDB-v3 exhibits substantially lower latency and can be deployed locally without requiring cloud connectivity. These results identify a quality-latency trade-off between direct large-model selection and clause-aware semantic retrieval for privacy-sensitive, connectivity-constrained, and resource-constrained robotic systems.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.