Can AI Scientists Coordinate at Runtime?
Current multi-agent AI scientists rely on fixed, design-time workflows, yet human scientists adapt their division of labor dynamically. This paper proposes Runtime Agent Coordination (RAC), which lets agents select the next capability at runtime based on research state, scoped work contracts, and artifact-grounded verification. Evaluated across three existing hosts on ResearchClawBench, the key finding is that runtime selection alone improves mean scores, but adding contracts and verification often hurts performance under tight budgets. The work is conceptually sound but empirically thin, resting on a single seed and tiny task samples.
The paper makes a timely conceptual distinction between communication and coordination in multi-agent AI scientists, but its empirical conclusions must be treated as preliminary. Runtime selection (R2) achieves the highest observed mean for every host, yet the full framework (R3) underperforms this simpler condition in all cases, and the evaluation relies on a single seed with as few as five tasks per host condition. The authors are commendably transparent about these limits, noting that their rankings are descriptive and do not establish robustness across independent runs.
The conceptual separation of runtime coordination (who acts next) from inter-agent communication (what is shared) is well-motivated and overdue. The cumulative ablation design—native execution (N0), plus communication (R1), plus runtime selection (R2), plus contracts and verification (R3)—provides a principled ladder for attributing effects. Engineering RAC as thin host bridges that preserve native agents, tools, and permissions is pragmatic and strengthens the claim that observed differences stem from coordination rather than implementation rewrites.
The evidence is severely underpowered. All runs use a single random seed, and ARK is evaluated on only five tasks (ten for the other two hosts), leaving no basis for variance estimation or statistical testing. More troublingly, contracts and verification (R3) uniformly reduce the mean relative to runtime selection alone (R2), suggesting that under the imposed budget ceilings these mechanisms add inference overhead without commensurate benefit. Budget matching is described as an intention rather than a verified fact: the archived configurations contain multiple ceiling settings, and the authors explicitly decline to assert that budgets were truly equalized across conditions.
The internal ablation structure is the strongest feature of the empirical design, but the small sample and absence of replication mean that even the consistent R2 advantage could be noise. Comparisons to related multi-agent systems (DyLAN, GPTSwarm, MetaGPT) are appropriately scoped, positioning RAC as a cross-host coordination layer rather than a competitor. The authors honestly note that because R3 jointly adds contracts and verification, their individual effects remain unresolved. The exploratory DiscoveryBench transfer test involves only six task-host pairs and is too fragmentary to support generalization claims.
Code is released, which aids reproducibility, but several critical details are missing or unresolved. Hyperparameters and exact prompts for the shared runtime selection policy $\pi_{\mathrm{RAC}}$ and the artifact verifier are not provided in the main text or appendices. Hop ceilings are calibrated per-host to native N0 counts, yet the authors concede that budget matching is an intended design rather than a verified equality of realized consumption. The single-seed, tiny-sample design precludes any estimate of run-to-run variance. Additionally, reliance on a future-dated pricing page for DeepSeek V4 Pro cost estimates and the use of a non-public judge model (GPT-5.5) may hinder exact independent reproduction.
Multi-agent AI scientists have shown improving performance across a diverse range of tasks. Yet a common approach is design-time agentic orchestration, which typically relies on fixed workflows. In contrast, human scientists coordinate and adjust their division of labor at runtime. We therefore ask: can AI scientists also coordinate at runtime? To this end, we introduce Runtime Agent Coordination (RAC), which selects agents from existing AI-scientist hosts during execution, assigns scoped work contracts, and provides artifact-grounded verification. Verification informs subsequent agents without blocking transitions or discarding artifacts. We conduct a single-seed exploratory evaluation across Agent Laboratory, EvoScientist, and ARK on ResearchClawBench, preserving host models, tools, and permissions under host-calibrated budgets. Four cumulative conditions separate native execution, runtime communication, runtime selection, and the combined addition of contracts and verification. Runtime selection yields the highest observed mean score for each host; adding contracts and verification reduces these means, with host-dependent outcomes relative to native execution. These results motivate runtime coordination while exposing the limits of additional coordination mechanisms under constrained budgets. Code is available at https://github.com/systemind-team/Runtime-AI-Scientist.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.