Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals
Peer review is buckling under submission volume, and LLM assistance is gaining traction. This paper asks how four current LLMs treat 200 already-published articles from top-tier journals such as Nature, Science, and Cell. The authors find that model recommendations are overwhelmingly harsh and starkly inconsistent: $605$ of $800$ reviews ($75.6\%$) recommended major revision or rejection, yet the rate for any single model ranged from $17.5\%$ to $100\%$ ($p < 0.001$). The takeaway is that LLM editorial signals are too variable to replace human judgment, though convergent criticisms may still be useful.
The paper presents a timely, well-designed benchmark showing that four current LLMs produce widely divergent and overwhelmingly harsh recommendations on published manuscripts, while largely ignoring deliberately planted flaws. The experimental design—using real published articles as a specificity benchmark and targeted error injection for sensitivity—is elegant. However, the manuscript is weakened by its reliance on other LLMs as quality judges rather than domain experts, the absence of a human baseline on the same corpus, and the selection of smaller, cheaper models that may not represent state-of-the-art capability. The core argument that LLM editorial recommendations constitute a weak and unstable signal is nevertheless well-supported by the evidence presented.
The strongest features are the methodologically sound experimental design and conservative statistical handling. Using 200 recently published articles from 20 journals as a collective ground truth for specificity is pragmatic and robust, and the paired comparison between first-submission and revision framings cleanly isolates prompt sensitivity. The analysis appropriately respects the ordinal nature of editorial decisions by employing Kruskal-Wallis and Friedman tests rather than assuming interval scales, and the authors commendably disclose their own use of AI in drafting and analysis.
The biggest limitation is the lack of a human baseline within the same corpus. While the authors rightly invoke the NeurIPS consistency experiments to note that human reviewers also disagree, they never run the same 200 manuscripts past human reviewers or compare LLM outputs to the actual referee reports these papers originally received, making it impossible to judge whether a $100\%$ major-revision rate from GPT-5.6 Sol is empirically worse than human performance on the same sample.
A second concern is the circularity of using LLMs to judge LLM reviews. The off-panel judges may share training-data biases and reward superficial fluency over domain depth, and no validation shows these judges correlate with expert human editors. The authors acknowledge that none of the reviews underwent expert assessment, referring to Kim et al. for such work, but this simply highlights that the present study does not close the evaluation loop. Additionally, the near-zero shift in recommendations even when a planted flaw is detected may reflect reasonable editorial conservatism rather than algorithmic indecision.
The evidence broadly supports the key claims, but comparisons to related work are incomplete. The paper fairly positions itself against Kim et al., who found that a GPT-5.2 agent scored above each paper's top-rated human reviewer on correctness, significance, and sufficiency ($60.0\%$ vs. $48.2\%$, $p = 0.009$), yet with much higher overlap between AI reviewers ($21\%$ vs. $3\%$). The present study does not replicate that human-expert assessment step, instead relying on external survey data and algorithmic judges for comparison. The statistical tests are well-chosen and reported with appropriate caution, and the claim that models exhibit low specificity and low sensitivity is a fair summary of the tabulated results, but without direct human benchmarking the leap from 'variable' to 'unacceptably unreliable' rests partly on the reader's prior expectations about human reviewer noise.
Reproducibility is mixed. The authors commit to depositing the corpus, flaw-injected variants, all review outputs, and analysis code in a permanent, DOI-assigned public repository prior to journal submission, and the exact prompts for every experimental condition are reproduced verbatim in Appendix A. On the negative side, the study uses non-deterministic models accessed via a commercial API, and there is no mention of temperature settings, seed values, or repetition parameters. The cost-driven model selection also means the results may not transfer to larger, more capable models, and the manuscript acknowledges potential training-data contamination without quantifying or controlling for it. Independent reproduction would be blocked primarily by commercial API costs and the lack of disclosed hyperparameters rather than by data availability.
With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.