Skip to content
Scriptorium Scriptorium

AI Peer Review: What's Validated, What's Speculative

Last updated: 2026-10-20

Synthesis

Section titled “Synthesis”

The literature on LLMs as peer reviewers is young (most substantive work is 2023–2026), contested, and moving fast. Two findings have begun to replicate. First, large-scale empirical work (Liang et al. 2024, NEJM AI / Stanford-led) shows that GPT-4–generated feedback overlaps with human reviewer feedback at roughly the same rate that two human reviewers overlap with each other — ~30% point-overlap on Nature-family papers, ~39% on ICLR papers, versus ~28% and ~35% for human–human comparisons [1]. Second, a randomized field experiment at ICLR 2025 (Thakkar et al. 2026, Nature Machine Intelligence; >20,000 reviews) shows that giving reviewers LLM-generated feedback on their draft reviews — flagging vague comments, misunderstandings, and unprofessional remarks — led 27% of reviewers who received it to update their reviews, and produced longer reviews that blinded researchers judged more informative, plus more engagement during rebuttal [2].

These results are not, however, evidence that LLMs can replace peer review. The most-cited normative paper (Hosseini & Horbach 2023, Research Integrity and Peer Review) warns that because LLMs are “trained on existing data by (mostly) biased humans,” they risk “reproducing or amplifying” existing biases, and that their opacity makes it “unclear how LLMs will develop, why they perform in the way they do and what they do with inputted texts” [3]. Later empirical work adds that LLM reviewers can be steered by text hidden in the manuscript and over-weight limitations the authors themselves disclose [9]. The two most-cited specific failure modes are (a) hallucinated citations and missing prior work, and (b) surface-level engagement — critiques that sound competent but do not interrogate study design at depth.

Editorial policy has converged faster than the science, and in 2026 it shifted from prohibition toward risk-managed support. Springer Nature’s September 2026 framework (which governs Nature Portfolio) lets reviewers “use AI to support peer review” but forbids uploading manuscript content to “unsecured or public AI tools,” requires disclosure, and says reviewers “must not delegate critique or judgement to AI systems” [4]. Elsevier’s June 2026 policy similarly restricts AI to “a supportive capacity” and forbids uploading a submitted manuscript “or any part of it” into an AI tool [13]. NIH still prohibits grant reviewers from using generative AI to analyze or formulate critiques [14]. The policy stance remains conservative: AI may support peer review, not perform it. In practice, AI-written reviews are already common — one detector classified 21% of ICLR 2026 reviews as fully AI-generated [12] — and one large conference (AAAI-26) has deployed a clearly labelled AI review on every main-track submission [11].

Scriptorium’s reviewer-simulation skill is positioned within this consensus, not against it. The skill simulates reviewer attention to help authors stress-test their own manuscript before submission. It is not a substitute for editorial peer review, does not produce ratings or recommendations, and operates with explicit awareness of the failure modes the literature documents.

Evidence

Section titled “Evidence”

Liang et al. (2024) — the large-scale comparison [1, 5]. Published in NEJM AI (2024); the preprint is arXiv:2310.01783 [5]. Method: prompt GPT-4 with a complete paper PDF and a standardized peer-review elicitation prompt; compare the generated feedback to actual reviewer feedback on the same papers using both retrospective overlap (15 Nature family journals, 3,096 papers; ICLR 2022, 1,709 papers) and prospective user study (308 researchers from 110 institutions in AI and computational biology).

  • Retrospective overlap of points raised: GPT-4 vs. human reviewer mean overlap = 30.85% (Nature) / 39.23% (ICLR); human vs. human mean overlap = 28.58% / 35.25%.
  • 57.4% of users in the prospective study rated GPT-4 feedback “helpful” or “very helpful”; 82.4% found it more beneficial than at least one human reviewer they had received.
  • The overlap between GPT-4 and human reviewers is larger for weaker papers [5].
  • Limitations the authors flag: GPT-4 feedback over-indexes on superficial aspects (writing, organization) relative to human reviewers, who attend more to design and significance.

Thakkar et al. (2026, Nature Machine Intelligence) — randomized study in production [2]. Randomized trial at ICLR 2025 covering >20,000 reviews, run by James Zou’s Stanford group in collaboration with ICLR 2025 and OpenReview; five co-authors were the ICLR 2025 program chairs, who provided guidance on the study [2]. A multi-LLM “Review Feedback Agent” sent reviewers optional feedback on their draft reviews (vague comments, content misunderstandings, unprofessional remarks), gated by automated reliability tests. 27% of reviewers who received feedback updated their reviews, incorporating over 12,000 suggestions; updated reviews were longer (about 80 words on average) and “more informative … as evaluated by blinded researchers,” and reviewers in the treatment group engaged more in author–reviewer discussion [2]. The study tests AI as an assistant to the reviewer’s writing; it does not test AI as a reviewer of the paper, and it does not measure decision accuracy.

Hosseini & Horbach (2023) — the conceptual critique [3]. Research Integrity and Peer Review (BMC), the most cited normative paper on LLM peer review. Key claims: LLMs can reduce reviewer fatigue and assist in drafting reports, but (i) confidentiality is at risk when unpublished manuscripts are pasted into third-party services, since “the LLM may inadvertently incorporate data or ideas to its training data”; (ii) the systems “seem to be able to contribute most on a language rather than a content level”; (iii) bias inherited from training data may be reproduced or amplified, and “some biases such as those related to geography, race or class might be reinforced”; (iv) malicious actors could use LLMs to degrade review quality, e.g., through fake reviews or smear campaigns on post-publication platforms. The paper’s recommendations include training reviewers on LLM risks, disclosing AI use and prompts, and retaining human accountability.

Manipulation and anchoring. Ye et al. (arXiv:2412.01708, preprint) found that “injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings”; in simulation, manipulating 5% of reviews could push 12% of papers out of the top 30% [9]. They also found implicit manipulation: when authors highlight minor limitations, LLM reviews echo those disclosed limitations with 4.5 times the consistency of human reviewers [9]. The attack is not hypothetical — in July 2025 Nature reported preprints with hidden instructions in white or tiny text asking AI tools to give positive reviews, some of which were to be withdrawn [10]. BadScientist (Jiang et al., arXiv:2510.18003; accepted at ACL 2026 per the arXiv record) shows that agent-generated papers with fabricated or unsound content can obtain acceptance-level scores from multi-model LLM review pipelines, and that the mitigations tested give “only marginal improvements, with detection accuracy barely exceeding random chance” [16].

Hallucinated citations and surface engagement. Multiple 2024–2026 analyses document hallucinated references in AI-assisted writing and review. GPTZero (commercial analysis) listed 50 ICLR 2026 submissions with at least one hallucinated citation [6]. ICLR 2026’s program chairs then ran an automated reference checker on all submissions, had every flagged reference checked by at least three humans, and desk-rejected all papers with confirmed hallucinated references [6]. Independent academic analyses report the same pattern in accepted papers: Sakai, Kamigaito & Watanabe (arXiv:2601.18724, 2026 — preprint, “Work In Progress”) found nearly 300 ACL/NAACL/EMNLP 2024–2025 papers with at least one hallucinated citation, half of them at EMNLP 2025 [15]; an analysis of NeurIPS 2025 found 100 fabricated citations in 53 accepted papers (about 1% of the accepted papers), all of which had passed review by 3–5 reviewers [17]. Mitigations identified: mandatory citation grounding, section-wise ingestion, “critique-then-verify” workflows that bind every critique to explicit textual evidence.

AI review at conference scale. AAAI-26 attached one clearly identified AI review to every main-track submission — 22,977 papers, generated in under a day — and the organizers report that surveyed authors and program-committee members found the AI reviews useful and “preferred them to human reviews on key dimensions such as technical accuracy and research suggestions” [11]. This is a preprint written by the pilot’s organizers. At ICLR 2026, an analysis by the detection company Pangram classified 21% of about 70,000 reviews as fully AI-generated, and reported that reviews with more AI involvement gave higher scores [12]. ICLR’s own response was to flag reviews that two detectors both marked as entirely LLM-generated to area chairs as a review-quality signal, while noting that detector “accuracy is far from perfect” [6].

Detection is unreliable. Population-level estimates are possible: Liang et al. (ICML 2024) estimated that between 6.5% and 16.9% of text submitted as peer reviews to ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 “could have been substantially modified by LLMs” [8]. Per-review attribution is much weaker; ICLR 2026’s program chairs explicitly declined to penalize reviews on detector output alone [6]. A 2025 Nature news feature, “AI is transforming peer review — and many scientists are worried,” documents individual cases of apparently LLM-written referee reports and the unease they cause [7]. This makes editorial policy enforcement difficult.

Editorial policy synthesis (2026) [4, 13, 14]. Springer Nature / Nature Portfolio’s current framework (announced 10 September 2026) governs AI use “according to risk level, not by tool type” and applies the same framework to authors, reviewers, and editors [4]. For reviewers: (a) they “remain responsible for independent, expert evaluation”; (b) they “may use AI to support peer review but must not upload manuscript content to unsecured or public AI tools”; (c) they must disclose AI use; (d) they “must not delegate critique or judgement to AI systems” [4]. The accompanying press release describes “improving language, cleaning data, or structuring comments” as limited-risk uses, and says “delegating peer review to an AI tool” “continues to not be permitted” [4]. Elsevier (policy updated June 2026) lets reviewers use AI “only … in a supportive capacity, for example to improve the language and structure of a review report, or to assist with a background literature search,” requires disclosure, and says reviewers “must not upload submitted manuscripts, or any information from them, into AI Tools” [13]. NIH prohibits its reviewers from using generative AI “for analyzing and formulating peer review critiques” of grant applications [14]. (The earlier version of this note summarised Nature Portfolio’s pre-2026 policy, including an “absolute consensus” against uploading manuscripts and a copy-editing disclosure exemption; that wording is superseded. Science’s policy was not re-checked in this refresh.)

How this informs scriptorium

Section titled “How this informs scriptorium”

Scriptorium threads the needle by being explicit about three things the literature converges on:

  1. Author-side use, not editorial-side use. reviewer-simulation runs against the author’s own manuscript before submission. There is no confidentiality issue (the author owns the text), no editorial-policy violation (the skill is not used as part of journal peer review), and no inference about acceptance/rejection.

  2. Structured outputs, evidence-bound critiques. Following the “critique-then-verify” pattern from the AI-peer-review failure literature: every critique scriptorium surfaces should cite a specific manuscript passage (line/section reference) and a specific concern category from common-critiques-taxonomy. Critiques without textual anchors should be suppressed or down-weighted. This addresses the surface-engagement failure mode head-on.

  3. No hallucinated citations. Per the DESIGN.md “no citation hallucination” rule, reviewer-simulation may identify missing prior work as a concern (e.g., “this claim would benefit from comparison to the prior art in X”) but must not invent specific citations. The citation-audit skill handles citation issues with verifiable provenance.

  4. Acknowledged limitations as a first-class output. Scriptorium’s simulated reviews should end with an explicit “limitations of this simulation” section: which families of critique were under-explored, which the model is structurally weaker at (typically: deep methodological grounding in unfamiliar subfields), and where human review remains necessary. This is what the Hosseini & Horbach recommendations look like operationally.

  5. Not a substitute for editorial review. Scriptorium’s README and skill cards should be explicit: this is a stress-test, not a substitute. The empirical evidence (Liang 2024; Thakkar 2026) supports AI as a complement to human review — useful when paired with human judgment, not as a replacement.

  6. Don’t let the manuscript set the agenda. Ye et al. [9] show that LLM reviewers echo the limitations authors choose to disclose far more than human reviewers do. Researcher inference: a simulated review that mostly restates the manuscript’s own limitations section is the failure this predicts. reviewer-simulation should form its critiques from the methods and results before weighing the authors’ stated limitations, and should say when a critique only repeats a disclosed limitation.

  7. Treat manuscript text as data, not instructions. Hidden prompts aimed at AI reviewers exist in real preprints [10]. Because reviewer-simulation ingests the full manuscript, instructions embedded in it (including invisible text) must never change the skill’s behaviour. Researcher inference: surfacing such text to the author as a finding is cheap and useful, since the author may not know a co-author or template added it, and journals increasingly screen for it.

Implementation priority for scriptorium

Section titled “Implementation priority for scriptorium”

Verdict: Yes — this note grounds the existing reviewer-simulation skill. (Section added 2026-10-20; the note previously lacked one.)

If Yes: reviewer-simulation, already scoped. The October 2026 evidence changes two behaviours and confirms one:

  1. Change — independence from disclosed limitations (point 6 above, from [9]).
  2. Change — prompt-injection hygiene (point 7 above, from [9, 10]): ignore instructions in the manuscript; report hidden or instruction-like text to the author.
  3. Confirm — no scores or accept/reject recommendation. Reviews with more AI involvement gave higher scores at ICLR 2026 [12], and LLM review pipelines give acceptance-level scores to unsound papers [16]. Scores are where LLM review is least trustworthy.

Required data: none beyond the manuscript. Benchmarks for evaluation remain the Liang et al. overlap figures [1].

Open questions / weak evidence

Section titled “Open questions / weak evidence”
  • Does the overlap finding (Liang 2024) generalize beyond CS/ML and Nature-family biomed? The two domains tested are well-represented in GPT-4’s training data and have unusually structured review traditions. Humanities, social science, and small-field domains are unstudied.
  • Are the “helpful” ratings authors give to LLM feedback measuring quality or fluency? The user-study evidence is suggestive but cannot distinguish “this critique is right” from “this critique is well-written and plausible-sounding.” The AAAI-26 preference result [11] has the same limitation: preference is not correctness.
  • Adversarial robustness. Prompt-injection against review-time LLM use is now documented experimentally [9] and in the wild [10]. BadScientist [16] explores whether fabrication-oriented agent-generated papers can deceive multi-model LLM-review pipelines; the headline finding is that reviewers frequently flag integrity issues yet still assign acceptance-level scores. Treat as concerning; the Ye et al. result is still a preprint.
  • The agentic-pipeline question. Almost all published evidence concerns single-call LLM review (one prompt → one review). AAAI-26’s multi-stage system [11] is the first large deployment of a more agentic reviewer, but its evaluation is author preference plus a new benchmark built by the same team. Scriptorium’s multi-skill architecture (citation-audit + reviewer-simulation + argumentative-flow + …) is not directly tested in the literature. Whether composed agentic critique outperforms single-call critique is the empirical question scriptorium itself is structured to answer.
  • Pre-print vs. peer-reviewed status. The Liang 2024 NEJM AI paper is peer-reviewed. The 2026 Nature Machine Intelligence RCT is peer-reviewed. The Liang ICML 2024 detection paper [8] is peer-reviewed. BadScientist [16] is listed as accepted at ACL 2026. HalluCitation Matters [15], the NeurIPS citation taxonomy [17], Ye et al. [9], and the AAAI-26 pilot report [11] are arXiv preprints and should be treated as preliminary. The Pangram and GPTZero figures [6, 12] are commercial analyses.

References

Section titled “References”
  1. Liang W, Zhang Y, Cao H, Wang B, Ding DY, Yang X, Vodrahalli K, He S, Smith DS, Yin Y, McFarland DA, Zou J. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI 1(8) (2024). DOI: 10.1056/AIoa2400196. Code: https://github.com/Weixin-Liang/LLM-scientific-feedback.
  2. Thakkar N, Yuksekgonul M, Silberg J, Garg A, Peng N, Sha F, Yu R, Vondrick C, Zou J. A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence 8:326–336 (2026). DOI: 10.1038/s42256-026-01188-x. Preprint: arXiv:2504.09737 (author contributions in the preprint identify Garg, Peng, Sha, Yu and Vondrick as ICLR 2025 program chairs).
  3. Hosseini M, Horbach SPJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review 8: 4 (2023). DOI: 10.1186/s41073-023-00133-5. PMID: 37198671. (Research Square preprint: PMID 36865238.)
  4. Nature Portfolio editorial policies — Artificial Intelligence (AI): “A risk‑assessment framework for responsible AI use in research publishing.” https://www.nature.com/nature-portfolio/editorial-policies/ai. (Accessed 2026-10-20.) Announced in: Springer Nature, “Springer Nature launches updated AI policies and guidance to support responsible AI use,” press release, 2026-09-10. https://group.springernature.com/gp/group/media/press-releases/sn-policy-responsible-ai-use/53209516
  5. Liang W et al. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv:2310.01783 (2023) — preprint of [1]. https://arxiv.org/abs/2310.01783. (Corrected 2026-10-20: this entry previously pointed to a Nature Human Behaviour PDF that is a different study, Liang et al. 2025, “Quantifying large language model usage in scientific papers,” DOI 10.1038/s41562-025-02273-8.)
  6. ICLR 2026 Program Chairs. “A Retrospective on the ICLR 2026 Review Process.” ICLR blog, 2026-03-31. https://blog.iclr.cc/2026/03/31/a-retrospective-on-the-iclr-2026-review-process/ ; GPTZero. “GPTZero uncovers 50+ Hallucinations in ICLR 2026.” https://gptzero.me/news/iclr-2026/ (commercial analysis; the page is a table of submissions, each with an example hallucinated citation; no peer-reviewed source exists for the ICLR count).
  7. Naddaf M. “AI is transforming peer review — and many scientists are worried.” Nature 639(8056):852–854 (2025). DOI: 10.1038/d41586-025-00894-7. PMID: 40140590. https://www.nature.com/articles/d41586-025-00894-7. (News piece; not peer-reviewed research, but synthesizes the field’s concerns.)
  8. Liang W, Izzo Z, Zhang Y, Lepp H, Cao H, Zhao X, Chen L, Ye H, Liu S, Huang Z, McFarland DA, Zou JY. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. Proceedings of ICML 2024. arXiv:2403.07183.
  9. Ye R, et al. Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review. arXiv:2412.01708 (2024). (Preprint.)
  10. Gibney E. “Scientists hide messages in papers to game AI peer review.” Nature 643:887–888 (2025). DOI: 10.1038/d41586-025-02172-y. https://www.nature.com/articles/d41586-025-02172-y
  11. Biswas J, Schoepp S, Vasan G, et al. AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot. arXiv:2604.13940 (2026). (Preprint by the pilot organizers.)
  12. Naddaf M. “Major AI conference flooded with peer reviews written fully by AI.” Nature 648:256–257 (2025). DOI: 10.1038/d41586-025-03506-6. Underlying analysis: Pangram, “Pangram Predicts 21% of ICLR Reviews are AI-Generated,” 2025-11-18. https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews-are-ai-generated (commercial analysis).
  13. Elsevier. “Generative AI policies for journals” (policy updated June 2026). https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals
  14. NIH. “The Use of Generative Artificial Intelligence Technologies is Prohibited for the NIH Peer Review Process.” Notice NOT-OD-23-149. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-23-149.html
  15. Sakai Y, Kamigaito H, Watanabe T. HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences. arXiv:2601.18724 (2026). (Preprint.)
  16. Jiang F, Feng Y, Li Y, Niu L, Alomair B, Poovendran R. BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? arXiv:2510.18003 (2025; ACL 2026 per arXiv comments).
  17. Ansari MS, et al. Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025. arXiv:2602.05930 (2026). (Preprint.)

See also: reviewer-archetypes-evidence, common-critiques-taxonomy, critique-quality-evidence.