Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings
This paper tackles validation congestion in scientific discovery, where enormous virtual libraries overwhelm scarce experimental capacity. Instead of relying on data-hungry regressors, it proposes PRISMS, a framework that curates candidates using multi-fidelity pairwise rankings aggregated through a regularized Bradley-Terry model, escalating queries to expensive experts via a Fisher-information criterion. The central claim is that comparative judgments from heterogeneous experts can slash the number of slow experimental rounds needed to find top-performing designs.
PRISMS is a well-motivated and conceptually elegant shift from absolute-score regression to pairwise-ranking-based curation. In simulated screening campaigns it reaches 50% top-10 recall in roughly 42% fewer rounds than GP-EI, and in multi-objective optimization it improves hypervolume by about 18.8% over a GP-BO baseline. These gains are internally validated by strong ablations. Yet the overall assessment is tempered by the paper's heavy reliance on simulated higher-fidelity rankers and very limited real-world validation; the headline estimate of saving '8--24 months' is extrapolated from an oracle setting rather than observed in practice.
The design of PRISMS is principled and its components are validated with care. The joint estimation objective anchored by a regularized Bradley-Terry model provides a statistically grounded way to fuse rankings with sparse measurements, and the ablation study shows that removing key pieces degrades performance predictably. Most convincingly, information-guided escalation outperforms random escalation, and adding a third fidelity stage improves AUDC@10 by 11.4%.
The largest concern is the simulation-to-reality gap. The higher-fidelity ranker is a simulated 75%-accurate oracle, and the only real-world proxy---an LLM-based agent evaluated on three tasks and three seeds---underperforms the oracle by a wide margin (AUC 0.105 versus 0.162). The paper also assumes a fixed, global fidelity ordering, which Appendix A acknowledges is unrealistic when expert accuracy varies by chemical scaffold or improves during a campaign. Finally, while runtime is modest versus physical experiments, scaling to 100,000 candidates takes over 3,200 seconds per round, which could limit interactive use.
Internal comparisons are fair among ranking methods because HodgeRank, RankRefine++, and PRISMS share identical ranker access and budgets, cleanly isolating the value of joint optimization and information-guided escalation. However, the regression baselines operate under a severe handicap. The GP-EI screening baseline begins with only 10 labels, a starvation regime where GPs are known to struggle, and the GP-BO optimization baseline is cited without details on prior width or acquisition tuning---precisely the pitfalls that Tripp and Hernandez-Lobato show can 'predictably' cripple BO performance. Consequently, the reported 42% and 18.8% improvements may shrink against stronger, tuned regression surrogates or modern preferential BO variants.
Reproducibility is hampered by the absence of any stated code release or persistent repository. While the datasets are drawn from cited public sources and Appendix C documents optimization settings such as L-BFGS-B tolerances, critical hyperparameters---including per-dataset values for the temperature $\tau$, ranker weights $\lambda_1^{(\ell)}$, and prior regularization $\lambda_2$---are not fully tabulated. Without open-source code and exact experimental scripts, independent reproduction of the 14-dataset screening benchmark or the LLM proof-of-concept would require substantial reimplementation.
Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a validation congestion where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on curation: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ~18.8% higher hypervolume than the Bayesian optimization baseline.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.