Which LLM to pick? Online Active Model Selection for Large Language Models
Selecting the best large language model for a live stream of prompts is difficult because benchmark scores are only coarse proxies for real-world performance and oracle annotations are costly. This paper proposes Online LLM Picker, the first active model selection framework designed specifically for LLMs in streaming settings: it uses an Exponential Weights posterior and an adaptive query probability based on hypothetical response variance scaled by posterior entropy to decide which prompts are worth annotating. Evaluated across over 130 models and 10 datasets spanning question answering, dialogue, grammar correction, and math, the method claims to cut annotation cost by up to 71.67% and reduce regret by up to 2.51× compared to baselines.
The paper tackles a timely and practically important problem—online model selection for generative LLMs—and proposes a principled algorithm that blends online learning with active sampling. Its scale is commendable, spanning 10 datasets and 130+ models, and the core query probability based on variance and posterior entropy is well-motivated. However, the work lacks formal regret or identification guarantees for the proposed heuristic query rule, and the headline savings hide dataset-level heterogeneity, including settings where the method is no better or worse than the strongest baseline.
The combined informativeness criterion is a clear conceptual strength: maximum hypothetical variance across model outputs captures disagreement on a given prompt, while normalized posterior entropy captures overall uncertainty about which model is best. The adaptive learning rate derived from an exponential moving average of observed loss variance is a pragmatic transfer of AdaHedge ideas to sparse-feedback settings, and the use of both ROUGE-L and BERTScore provides lexical and semantic coverage of output quality. The breadth of tasks—open-ended QA, grammar correction, arithmetic, calculus, and dialogue—demonstrates that the framework is genuinely task- and model-agnostic.
The paper provides no theoretical characterization of the proposed query probability $q_t$: because $q_t$ appears in the denominator of the importance-weighted loss estimator, the variance–entropy heuristic can in principle lead to unstable estimates, yet no regret or query-complexity bound is given. The evaluation also hinges entirely on automatic metrics (ROUGE-L and BERTScore) without validating that the selected near-best model under these metrics is actually preferred by humans or an LLM judge, which is risky when these metrics are known to correlate imperfectly with human quality judgments.
Reproducibility details are sparse in the main text: the EMA decay $\\beta$, the number of independent stream realizations, and the exact annotation budgets per dataset are omitted or relegated to appendices. The baseline tuning procedure—scaling query probabilities by ad-hoc factors to match average budgets—is underspecified, and the claim that 130+ models were evaluated obscures that many are prompt-engineered variants of the same underlying LLM. Notably, Table 1 reveals weak spots: Grammar Correction shows 0% savings at most thresholds, and the Calculus dataset requires 28% more budget than the best baseline at $\\delta=2.5\\%$.
The evidence broadly supports the headline claim that Online LLM Picker reduces annotation cost, but the gains are uneven. Figure 3 shows the method consistently reaching 100% identification probability before baselines, yet Table 1 demonstrates that on some datasets the advantage vanishes or reverses. Comparisons to related work are fair and accurate: the authors correctly distinguish Ashury-Tahan et al. as pool-based pairwise selection for text generation, and they note that Karimi et al. and Okanovic et al. are limited to classification or pool-based settings. Still, the absence of a human or LLM-judge validation bottleneck means the paper optimizes surrogate metrics rather than true oracle utility.
The authors claim complete reproducibility and promise that complete source code and scripts are included as supplementary material, which is a strong commitment. However, the main text omits several details that would be essential for independent reproduction: the EMA decay rate $\\beta$ in the adaptive learning rate, the number of independent realizations averaged for each reported point, and the per-dataset annotation budgets are buried in appendices. The hyperparameter tuning strategy—multiplying query probabilities by hand-tuned scaling factors to align average annotation counts—is described only at a high level, and there is no discussion of inference cost or wall-clock time for running 130+ models across all realizations.
Large Language Models (LLMs) are increasingly applied to process streaming data, with practitioners relying on benchmarks to select the best model even though these signals only approximate real performance. While oracle annotations can provide reliable feedback, they are often costly and difficult to obtain at scale. To address this challenge, we propose ONLINE LLM PICKER, the first framework for active model selection for LLMs in online settings. Given an arbitrary stream of queries and a limited annotation budget, ONLINE LLM PICKER selects the most informative prompts for annotation to identify the best LLM among candidate models. Across multiple tasks including 10 datasets, for over 130 language models, we show that ONLINE LLM PICKER saves annotation cost by up to 71.67% while reliably identifying the best or near-best model for the stream. We also show that using the returned model for sequential generation on unannotated prompts across the stream reduces regret by up to a factor of 2.51x, indicating that ONLINE LLM PICKER can identify the best or near-best model well before processing all streaming prompts.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.