Agentic AI for scientific writing and discovery

Last updated: 2026-10-20

Synthesis

The agentic-AI-for-science landscape circa mid-2026 is bifurcated. On one side are fully-autonomous research systems (Sakana’s AI Scientist, FutureHouse’s Crow/Falcon/Owl/Phoenix and its Kosmos successor, Stanford’s CellVoyager) that aim to do science end-to-end: hypothesis, code, experiment, manuscript, review. On the other are single-task agents (AutoBA for bioinformatics pipelines, GeneAgent for gene-set annotation, ChemCrow/Phoenix for synthesis planning) that wrap LLMs around domain APIs to make one capability more reliable. The first group makes large claims that are only partially validated; the second group makes narrower claims that hold up better in benchmarking.

What is conspicuously absent from this landscape is anything that treats manuscript improvement as the unit of work. The fully-autonomous systems generate manuscripts as a side-effect of doing research; they do not iteratively improve a manuscript that a human has drafted. The single-task agents do not concern themselves with prose at all. The closest neighbor is the AI-peer-review work (Liang et al., AgentReview, OpenReviewer, Stanford’s PaperReview.ai), which produces critique but does not coordinate revision.

Scriptorium occupies that gap. It is not trying to write papers from scratch and it is not trying to replace peer review. It is trying to be the operating layer that sits between the human author and the agents, with MANUSCRIPT_STATE.yaml as the editorial source of truth — the thing none of the above systems have.

The October 2026 refresh adds two developments that sharpen this positioning. First, the autonomous systems have moved into high-profile venues (Nature for the AI Scientist [3], Science for Biomni [20], Nature Methods for CellVoyager [8]). The AI Scientist paper itself is candid that the peer-review bar it cleared was a workshop-level one, and that humans filtered its outputs along the way [3]. Second, independent work now shows that autonomous pipelines hide methodological failures that are invisible in the final paper and detectable only from trace logs and code [27] — a direct argument for keeping the human author, and an inspectable editorial record, in the loop.

Landscape

Fully-autonomous research / writing systems

Sakana AI Scientist (Lu et al., 2024) — The most-cited end-to-end system. The original arXiv paper (arXiv:2408.06292) [1] proposed a pipeline that generates ideas, writes code, runs experiments, produces a manuscript, and runs a simulated review. The GitHub repo SakanaAI/AI-Scientist has 14,765 stars and was last pushed 2025-12-19 [2]. A v2 successor (SakanaAI/AI-Scientist-v2, 7,377 stars, last pushed 2025-12-19; arXiv:2504.08066 [18]) was released, and a paper describing the system, “Towards end-to-end automation of AI research,” was published in Nature in March 2026 [3]. The Nature paper reports that a manuscript generated by the system “passed the first round of peer review for a workshop of a top-tier machine learning conference,” a workshop with a 70% acceptance rate [3]. The paper’s own limitations section notes that only one of three submissions was accepted and contrasts the 70% workshop acceptance rate with 32% for the ICLR 2025 main conference [3]. The process was not hands-off: the authors write that they “manually filtered the most promising outputs at each stage” before three manuscripts were selected for submission, and, under a protocol fixed in advance, all AI-generated submissions were withdrawn after review regardless of outcome, so the accepted paper was never published [3]. The v2 preprint nonetheless describes the submissions as “three fully autonomous manuscripts” [18].

The independent evaluation by Beel et al. (arXiv:2502.14297; SIGIR Forum 2025) [4] is unsparing: 42% of experiments failed due to coding errors; literature reviews used simplistic keyword search and misclassified established concepts (e.g., micro-batching for SGD) as novel; generated papers had a median of 5 citations; outputs contained structural errors including missing figures and placeholder text like “Conclusions Here”. Cost was the headline finding: complete papers for $6–15, with 3.5 hours of human involvement. Sakana addressed many critiques in v2 by removing the human template dependency [18].

Hidden pitfalls of autonomous pipelines (Luo, Kasirzadeh & Shah, 2025/2026) — A controlled evaluation of two open-source AI-scientist systems that isolates four failure modes: inappropriate benchmark selection, data leakage, metric misuse, and post-hoc selection bias [27]. The authors found several of these failures, “which can be easily overlooked in practice,” and showed that trace logs and code from the full workflow make failures far easier to detect than the final paper alone. They recommend that venues evaluating AI-generated research require these artifacts alongside the paper [27].

FutureHouse platform (Crow, Falcon, Owl, Phoenix) — Launched 2025-05-01 by the philanthropically funded research lab FutureHouse, founded by Sam Rodriques and Andrew White [5, 7]. Crow is the production version of the open-source PaperQA2 (Future-House/paper-qa, 9,424 stars, last pushed 2026-10-20 [6]) and is a literature-QA agent. Falcon is the deep-review agent with access to specialized databases (OpenTargets, etc.). Owl (“HasAnyone”) answers “Has anyone done X before?”. Phoenix is an evolution of ChemCrow for chemistry. FutureHouse claims Crow/Falcon/Owl have been “experimentally validated” as having “better precision than PhD-level researchers in head to head literature search tasks” [5]. The underlying evidence is the PaperQA2 paper [19]: on the LitQA2 benchmark PaperQA2 reached 85.2% precision and 66.0% accuracy, against 73.8% precision and 67.7% accuracy for human annotators — “superhuman precision” but no significant difference in accuracy [19]. The platform is free with API access.

Kosmos (FutureHouse / Edison Scientific, 2025) — An “AI scientist” that runs for up to 12 hours of cycles of parallel data analysis, literature search, and hypothesis generation, coordinated through a structured “world model,” and writes reports in which every statement is cited to code or primary literature [25]. Independent scientists judged 79.4% of statements in Kosmos reports accurate [25]. This is a preprint (arXiv:2511.02824) and the accuracy figure comes from the developers’ own evaluation design. Its relevance to scriptorium is the shared-state idea: Kosmos treats a persistent world model as the thing that keeps long agent runs coherent, the same role MANUSCRIPT_STATE.yaml plays for editorial work.

CellVoyager (Alber, Chen, Sun, Isakova, Wilk, Zou; Stanford) — Published in Nature Methods (online 2026-03-17; 23(4):749–759 [8]; bioRxiv 2025-06-03). An LLM-based agent that autonomously explores scRNA-seq datasets in a live Jupyter environment, generating and refining “exploration blueprints.” On CellBench (76 published scRNA-seq studies) it is reported to outperform GPT-4o and o3-mini by up to 23% in predicting which analyses the original authors conducted, given only the papers’ background sections. In three case studies (COVID-19, cell–cell communication, aging), expert reviewers consistently rated CellVoyager-generated findings as creative and scientifically sound.

Single-task biomedical / bioinformatics agents

AutoBA (Zhou et al., 2024, KAUST) — Published in Advanced Science 11(44) [9]. An agent that takes three inputs (data path, data description, objective) and autonomously produces analysis plans, code, and execution. Supports WGS/WES, ChIP-seq, RNA-seq, scRNA-seq, spatial transcriptomics. Has an automated code repair (ACR) mechanism. Repo: JoshuaChou2018/AutoBA, 240 stars, last pushed 2024-11-04 [10]. Activity has slowed since publication — a useful warning sign about post-paper sustainability.

GeneAgent (Wang et al., NIH NLM) — Published in Nature Methods (August 2025, 22:1677–1685 [11], arXiv:2405.16205 preprint). A self-verification language agent that interacts with 18 biomedical databases via 4 Web APIs to verify its own gene-set annotations. Four-stage pipeline: generate, self-verify, modify, summarize. On 1,106 gene sets, GeneAgent was “consistently more accurate than GPT-4 by a significant margin.” Repo: ncbi-nlp/GeneAgent, 120 stars, last pushed 2025-12-25.

Biomni (Stanford) — General-purpose biomedical AI agent integrating 105 software tools, 150 specialized biological tools, 59 databases (bioRxiv 2025.05.30 [12]). Benchmarked on causal gene prioritization, drug repurposing, rare disease diagnosis, microbiome analysis, molecular cloning. Available at biomni.stanford.edu. The peer-reviewed version, “Autonomous biomedical research with an artificial intelligence agent,” appeared in Science in 2026 [20].

ChemCrow / Phoenix — Tool-using LLM for synthesis planning, evolved into FutureHouse Phoenix. The benchmarking is thinner than the other FutureHouse agents; FutureHouse itself says Phoenix “is not as deeply benchmarked as our other agents and may make more mistakes” [5].

AI-peer-review systems (closest neighbors to scriptorium’s reviewer-simulation)

Liang et al. — The first study (NEJM AI, arXiv:2310.01783) compared GPT-4 feedback to human reviewers on 3,096 Nature-family papers and 1,709 ICLR papers; GPT-4–to-human overlap was 30.85% (Nature) and 39.23% (ICLR), comparable to human-to-human overlap of 28.58% and 35.25% respectively; 57.4% of 308 surveyed researchers found GPT-4 feedback helpful, 82.4% found it more useful than some human reviewers [13]. A follow-up randomized study from the same Stanford group (Thakkar et al.; Liang is not an author) was published in Nature Machine Intelligence (8:326–336, 2026) [14]. It gave LLM-generated feedback on reviewers’ draft reviews at ICLR 2025, not LLM reviews of papers.

Stanford PaperReview.ai — Agentic system grounded in arXiv literature. Reports a Spearman correlation of 0.42 between AI score and human score, versus 0.41 between two human reviewers, on a held-out set of 147 ICLR 2025 submissions [15].

AgentReview — Multi-agent simulation of peer review with reviewers, area chairs, and authors as distinct LLM agents (EMNLP 2024) [16, 22]. ReviewerToo — Validated on 1,963 ICLR 2025 papers with 81.8% binary accept/reject accuracy, against 83.9% for the average human reviewer (arXiv:2510.08867, preprint) [21]. OpenReviewer (maxidl/openreviewer) — A specialized LLM for generating critical reviews (arXiv:2412.11948; NAACL 2025 System Demonstrations).

Agents4Science (2025) — Billed by its organizers as “the first conference in which AI agents serve as both primary authors and reviewers, with humans as co-authors and co-reviewers” [26]. The organizers’ report is in Nature Biotechnology [26]. The conference site says disclosures of AI involvement are “to be released to the public” and that the organizers “provide the prompts and reviews generated by AI review agents, serving as an open resource to the community” [31]. Researcher inference: that makes it a useful public record of AI-authored papers and their AI reviews; it is not evidence that AI review can replace human review.

Publisher-side integrity tools — IOP Publishing’s Duplicate Review Checker, reported by Nature in May 2026 as the “first AI tool to detect suspicious peer reviews rolled out by academic publisher,” flags reviewer reports that closely match other reports; IOPP reports that it processed about half a million reports from 2020 onward and found nearly 2,500 cases with more than 60% matching content [17, 23]. Separately, Springer Nature launched an in-house AI tool to flag irrelevant references in submitted manuscripts in April 2025 [24].

General-purpose “research assistant” agents

Genspark Super Agent — Multi-model “super agent” that produces Sparkpages (dynamically synthesized research documents). Marketed broadly; no peer-reviewed evaluation of research-quality outputs.

SciSpace Agent — Bills itself as uniting 150+ academic tools (covered in ai-writing-tools-survey).

Jenni — Workspace-style writing assistant with citation tracing.

What “claims” vs. “demonstrated outputs” looks like across this space

System	Headline claim	Demonstrated output
Sakana AI Scientist v1/v2	Fully autonomous discovery	One of three submissions passed first-round review at a workshop with a 70% acceptance rate, after humans filtered outputs at each stage; accepted paper withdrawn under a pre-set protocol; 42% experiment failure rate; thin citations
FutureHouse Crow/Falcon	”Better than PhDs” on LitQA	Higher precision than human annotators (85.2% vs. 73.8%) but no difference in accuracy (66.0% vs. 67.7%) on LitQA2; no end-to-end paper evidence
Kosmos	Autonomous data-driven discovery	79.4% of report statements judged accurate; preprint, developer-designed evaluation
CellVoyager	Novel scientific findings	3 case studies rated creative by experts; 23% improvement on author-prediction proxy
AutoBA	Fully automated multi-omic analysis	Works across many pipelines; activity has stalled post-publication
GeneAgent	Reduced hallucinations via DB verification	Significant accuracy gain over GPT-4 on 1,106 gene sets
AgentReview / OpenReviewer	Simulated peer review	Useful for studying review dynamics; not yet a substitute

How this informs scriptorium

• Scope discipline. None of the fully-autonomous systems coordinates iterative improvement of a human-authored manuscript. Scriptorium’s positioning is “operating system between author and agents,” not “another autonomous researcher.” This is a genuinely empty seat.
• Single-responsibility as a design principle, not just a preference. AutoBA’s slowing activity and Sakana’s 42% experiment-failure rate are warnings: wide-scope agents are hard to maintain and harder to trust. Scriptorium’s leaf skills (citation-audit, reviewer-simulation, argumentative-flow) are a deliberate bet that narrow + composable beats wide + autonomous.
• reviewer-simulation is in a contested arena. Liang et al.’s 30.85% overlap result is the baseline this skill should be benchmarked against. PaperReview.ai’s 0.42 Spearman is a closer comparison. Scriptorium should publish how its four-persona output compares to single-LLM review.
• Domain databases ground hallucinations. GeneAgent’s self-verification pattern (generate → check against DB → modify → summarize) is directly transferable to citation-audit: every citation claim should be verifiable against Crossref/PubMed metadata before it lands in the audit report.
• Manuscript state is the missing primitive. FutureHouse has agents but no shared editorial state. Sakana has a pipeline but no editorial state. The scriptorium thesis — that MANUSCRIPT_STATE.yaml is the gravity well — is testable against this absence. Kosmos’s “world model” [25] is the nearest analog, but it holds research state, not editorial state.
• Inspectable intermediate artifacts matter more than polished output. Luo et al. [27] show that failures in autonomous pipelines are visible in trace logs and code but not in the final paper. Researcher inference: the same logic supports scriptorium’s structured outputs and recorded editorial state — a reviewer of an AI-assisted revision needs the record of what changed and why, not just the revised text.

Implementation priority for scriptorium

Verdict: No new skill. Positioning and framing context.

Why useful context anyway:

1. Defends the scope. The landscape confirms that no system, including the 2026 Nature/Science publications, treats a human-drafted manuscript as the unit of work. This is the argument for scriptorium’s non-goal of autonomous research.
2. Supplies benchmarks for existing skills. reviewer-simulation should be evaluated against the Liang et al. overlap figures [13] and the PaperReview.ai correlation [15]; citation-audit borrows GeneAgent’s verify-against-database loop [11].
3. Supports the structured-output and editorial-state design. Luo et al. [27] is now the citable evidence that inspectable intermediate artifacts catch failures that final text hides.

Condition that would flip to Yes: a public benchmark of manuscript-before/after revisions appears (or scriptorium builds one), at which point an evaluation harness becomes a concrete deliverable rather than framing.

Open questions / weak evidence

• No published evaluation of any “manuscript improvement” agent on real revisions. All evaluations to date measure either generation quality or critique overlap, not whether agent-assisted revisions actually result in better papers (by reviewer scores, acceptance rates, citation counts). This is the evaluation scriptorium should aim to produce.
• The “AI peer reviewer” question is unresolved as a policy matter, but publisher policies converged in 2026. Elsevier’s policy (updated June 2026) allows reviewers to use AI only “in a supportive capacity,” forbids uploading a submitted manuscript “or any part of it” into an AI tool, and requires disclosure [28]. Springer Nature’s September 2026 risk-based framework lets reviewers use AI to support review but not upload manuscript content to “unsecured or public AI tools” or “delegate critique or judgement to AI systems” [29]. NIH prohibits reviewers from using generative AI to analyze or formulate critiques of grant applications [30]. Scriptorium’s reviewer-simulation is for self-use by authors, not for use as peer review — that distinction needs to remain crisp in documentation. See peer-review-platforms.
• Post-paper sustainability is the dominant failure mode. AutoBA’s activity dropped sharply after publication; meta-review (the manubot manuscript itself) has not been updated since 2020. Scriptorium needs a story for how it stays alive past v0.1.
• No standard benchmark for manuscript-improvement agents exists. LitQA exists for retrieval; ICLR papers exist for review; there is no equivalent for “manuscript before vs. after.” Building one would itself be a contribution.
• Developer-run evaluations dominate. The Kosmos accuracy figure [25], the PaperQA2 comparison [19], and the AI Scientist workshop result [3] were all designed by the system builders. Independent evaluations (Beel et al. [4], Luo et al. [27]) are consistently less favorable.

References

1. Lu C, Lu C, Lange RT, Foerster J, Clune J, Ha D. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” arXiv:2408.06292 (2024). https://arxiv.org/abs/2408.06292
2. SakanaAI/AI-Scientist GitHub. https://github.com/SakanaAI/AI-Scientist (14,765 stars; pushedAt 2025-12-19; checked 2026-10-20).
3. Lu C, Lu C, Lange RT, Yamada Y, Hu S, Foerster J, Ha D, Clune J. “Towards end-to-end automation of AI research.” Nature 651(8107):914–919 (2026). DOI: 10.1038/s41586-026-10265-5. PMID: 41882133. Full text: PMC13017497. Sakana announcement: https://sakana.ai/ai-scientist-nature/
4. Beel J, Kan M-Y, Baumgart M. “Evaluating Sakana’s AI Scientist: Bold Claims, Mixed Results, and a Promising Future?” SIGIR Forum 59(1):1–20 (2025). DOI: 10.1145/3769733.3769747. arXiv:2502.14297. https://arxiv.org/abs/2502.14297
5. FutureHouse Platform launch. https://www.futurehouse.org/research-announcements/launching-futurehouse-platform-ai-agents
6. Future-House/paper-qa GitHub. https://github.com/Future-House/paper-qa (9,424 stars; pushedAt 2026-10-20; checked 2026-10-20).
7. MIT News. “Accelerating scientific discovery with AI.” 2025-06-30. https://news.mit.edu/2025/futurehouse-accelerates-scientific-discovery-with-ai-0630
8. Alber S, Chen B, Sun E, Isakova A, Wilk AJ, Zou J. “CellVoyager: AI CompBio agent generates new insights by autonomously analyzing biological data.” Nature Methods 23(4):749–759 (2026). DOI: 10.1038/s41592-026-03029-6. PMID: 41845065. https://www.nature.com/articles/s41592-026-03029-6
9. Zhou J, et al. “An AI Agent for Fully Automated Multi-Omic Analyses.” Advanced Science 11(44) (2024). DOI: 10.1002/advs.202407094
10. JoshuaChou2018/AutoBA GitHub. https://github.com/JoshuaChou2018/AutoBA (240 stars; pushedAt 2024-11-04; checked 2026-10-20).
11. Wang Z, et al. “GeneAgent: self-verification language agent for gene-set analysis using domain databases.” Nature Methods 22:1677–1685 (2025). DOI: 10.1038/s41592-025-02748-6. PMID: 40721871. arXiv:2405.16205.
12. Biomni team. “Biomni: A General-Purpose Biomedical AI Agent.” bioRxiv 2025.05.30.656746. https://www.biorxiv.org/content/10.1101/2025.05.30.656746v1
13. Liang W, et al. “Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis.” NEJM AI 1(8) (2024). DOI: 10.1056/AIoa2400196. arXiv:2310.01783.
14. Thakkar N, Yuksekgonul M, Silberg J, et al. “A large-scale randomized study of large language model feedback in peer review.” Nature Machine Intelligence 8:326–336 (2026). DOI: 10.1038/s42256-026-01188-x
15. Stanford PaperReview.ai. https://paperreview.ai/tech-overview
16. AgentReview project. https://agentreview.github.io/
17. Naddaf M. “First AI tool to detect suspicious peer reviews rolled out by academic publisher.” Nature news, 2026-05-06. DOI: 10.1038/d41586-026-01454-3. https://www.nature.com/articles/d41586-026-01454-3
18. Yamada Y, Lange RT, Lu C, Hu S, Lu C, Foerster J, Clune J, Ha D. “The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search.” arXiv:2504.08066 (2025). https://arxiv.org/abs/2504.08066
19. Skarlinski MD, Cox S, Laurent JM, Braza JD, Hinks M, Hammerling MJ, Ponnapati M, Rodriques SG, White AD. “Language agents achieve superhuman synthesis of scientific knowledge.” arXiv:2409.13740 (2024). https://arxiv.org/abs/2409.13740
20. Huang K, Zhang S, Wang H, et al. “Autonomous biomedical research with an artificial intelligence agent.” Science 393(6813):eadz4351 (2026). DOI: 10.1126/science.adz4351
21. ReviewerToo. “ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review.” arXiv:2510.08867 (2025). https://arxiv.org/abs/2510.08867
22. Jin Y, Zhao Q, Wang Y, Chen H, Zhu K, Xiao Y, Wang J. “AgentReview: Exploring Peer Review Dynamics with LLM Agents.” Proceedings of EMNLP 2024. arXiv:2406.12708. https://aclanthology.org/2024.emnlp-main.70/
23. IOP Publishing. “New IOP Publishing tool detects duplicate peer reviews in push against reviewer fraud.” 2026-05-05. https://ioppublishing.org/news/new-iop-publishing-tool-detects-duplicate-peer-reviews-in-push-against-reviewer-fraud/
24. Springer Nature. “New research integrity AI tool added to Springer Nature’s growing portfolio.” Press release, 2025-04-07. https://group.springernature.com/gp/group/media/press-releases/new-research-integrity-ai-tool/27769148
25. Mitchener L, Yiu A, Chang B, et al. “Kosmos: An AI Scientist for Autonomous Discovery.” arXiv:2511.02824 (2025). https://arxiv.org/abs/2511.02824
26. Bianchi F, Queen O, Thakkar N, Sun E, Zou J. “Exploring the use of AI authors and reviewers at Agents4Science.” Nature Biotechnology 44(1):11–14 (2026). DOI: 10.1038/s41587-025-02963-8. arXiv:2511.15534.
27. Luo Z, Kasirzadeh A, Shah NB. “The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists.” PNAS 123(41):e2610214123 (2026). DOI: 10.1073/pnas.2610214123. Preprint: arXiv:2509.08713.
28. Elsevier. “Generative AI policies for journals” (policy updated June 2026). https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals
29. Nature Portfolio. “Artificial Intelligence (AI)” editorial policy (Springer Nature risk-assessment framework); announced in Springer Nature press release of 2026-09-10. https://www.nature.com/nature-portfolio/editorial-policies/ai ; https://group.springernature.com/gp/group/media/press-releases/sn-policy-responsible-ai-use/53209516
30. NIH. “The Use of Generative Artificial Intelligence Technologies is Prohibited for the NIH Peer Review Process.” Notice NOT-OD-23-149. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-23-149.html
31. Agents4Science 2025 conference site (“Open Conference of AI Agents for Science 2025”). https://agents4science.stanford.edu/ (accessed 2026-10-20).