Critique Quality: What Makes Review Feedback Actually Useful

Last updated: 2026-10-20

Synthesis

“Does peer review work?” has a more discouraging empirical answer than most researchers assume. The Cochrane systematic review on the value of peer review for biomedical journals (Jefferson et al., updated through several editions) concluded in its 2007 edition that “little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research” [1]; the 2003 edition was reported as stating that “the practice of peer review is based on faith in its effects, rather than on facts” [7]. That is — review exists, review is widely defended, but the controlled evidence that the process improves what gets published is thin.

The evidence that does exist points to two narrower, more defensible claims. First, peer review does improve manuscripts on specific dimensions, particularly around limitations, conclusions, and reporting (Goodman, Berlin et al. 1994 at Annals of Internal Medicine: 33 of 34 quality items improved post-review, with the largest gains in “discussion of study limitations, generalizations, use of confidence intervals, and the tone of conclusions”) [2]. Second, peer reviewers — even trained ones — detect only a fraction of major errors. Schroter et al. (2008) inserted nine major methodological errors into manuscripts sent to 607 BMJ reviewers; reviewers detected an average of 2.58 of 9 at baseline, and short training packages had “only a slight impact” on error detection [3].

What does this mean for scriptorium? Two things. (1) The benchmark a simulated reviewer should be compared against is not “perfect review” — it is “real human reviewers, who miss most major errors.” A simulation that surfaces methodological concerns the author would otherwise have missed adds value even if it is not exhaustive. (2) The dimensions on which peer review has demonstrable effect — limitations, generalizability, tone of conclusions, reporting quality — are exactly the dimensions on which structured AI critique can plausibly contribute, because they are the dimensions for which checklists, reporting guidelines (CONSORT, STROBE, ARRIVE), and the EQUATOR framework have made expectations explicit [4]. One caution: in the only randomized test we found, simply suggesting reporting checklists to reviewers did not measurably improve manuscript quality, whereas adding a statistical reviewer did [9].

Open peer review is the area where the critique-quality evidence is reassuring, though more mixed than often summarized. In the BMJ randomized trial (van Rooyen et al. 1999), asking reviewers to be identified had no important effect on review quality, recommendation, or time taken, but made reviewers more likely to decline (35% vs. 23%) [11]. In a British Journal of Psychiatry randomized trial (Walsh et al. 2000), signed reviews were of higher quality and more courteous, took longer to write, and were more likely to recommend publication [10]. A language-use comparison of closed review (Angewandte Chemie International Edition) with public review (Atmospheric Chemistry and Physics) found public-review comments much longer — which the authors read as a stronger “improvement function” — while closed-review comments actually used positive-emotion words more often [5]. The defensible generalization for scriptorium is narrower than “openness produces better critique”: making critique inspectable and attributable does not degrade it, and in one trial improved it — a reasonable basis (researcher inference) for treating inspectability as a design principle for agents.

Evidence

Does peer review catch real flaws? Mostly no.

Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 101(10): 507–514 (2008). Randomized trial with 607 BMJ reviewers (face-to-face training, self-taught package, or control) and three test papers each containing 9 major + 5 minor inserted errors. Baseline detection: 2.58/9 (~29%) major errors; means on the second and third papers were 2.71 and 3.0, and the interventions had small effects. Biased randomization was the error detected most frequently, identified by over 60% of the reviewers who rejected the papers; reviewers who did not reject found fewer errors, and fewer than 40% of them identified biased randomization. The authors conclude editors “should not assume that reviewers will detect most major errors” [3].
Jefferson T et al. (Cochrane). “Editorial peer review for improving the quality of reports of biomedical studies.” Multiple updates; the 2007 edition included 28 studies. Headline: little empirical evidence that peer review ensures quality, but “the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence.” Checklists and other standardisation media had “some evidence” in their favour (2 studies); there was no evidence that referee training affects outcome quality (1 study) [1].

Does peer review improve manuscripts? Yes, on specific axes.

Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 121(1): 11–21 (1994). 111 manuscripts assessed pre- and post-review on a 34-item quality instrument; 33/34 items improved. Largest improvements in discussion of limitations, generalizations, use of confidence intervals, and tone of conclusions. Manuscripts in the bottom 50% pre-review showed 2-3× larger improvements [2].
Cobo E et al. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS ONE 2(3): e332 (2007). 2×2 factorial trial at Medicina Clínica (115 manuscripts analysed). Adding a statistical reviewer improved quality on the Goodman scale (estimated effect 5.5 points, 95% CI 4.3–6.7); suggesting reporting guidelines (CONSORT, STARD) to clinical reviewers had no statistically significant effect (0.9, 95% CI −0.3 to 2.1) [9].

Open review changes critique quality — modestly.

Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review: a randomised controlled trial. Br J Psychiatry 176: 47–51 (2000). 408 manuscripts randomised to signed or unsigned review among reviewers who agreed to sign (76% agreed). Signed reviews were of higher quality, more courteous, and took longer to complete; reviewers who signed were more likely to recommend publication [10].
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 318: 23–27 (1999). Asking reviewers to be identified had no significant effect on review quality, publication recommendation, or time taken, but increased the likelihood of declining to review (35% vs. 23%) [11].
Bornmann, Wolf & Daniel (2012, Scientometrics) “Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?” used LIWC on 599 comments on 229 manuscripts from Angewandte Chemie International Edition (closed review) and Atmospheric Chemistry and Physics (public review). Public-review comments were much longer; closed review was not more negative in emotional tone, and positive-emotion words were significantly more frequent in closed review [5].

Reviewer training has small, short-lived effects.

Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: randomized controlled trial. BMJ 328: 673 (2004). PMID: 14996698. Two training interventions (face-to-face workshop, self-taught package) produced statistically detectable but small improvements in review quality and major-error detection; the quality difference “was not of editorial significance,” and the evidence of benefit was no longer apparent six months later. Trained reviewers were more likely to recommend rejection [6].
The Cochrane review likewise found no evidence that referee training improves outcome quality, on the basis of one study [1]. The disappointing-but-honest implication (researcher inference): reviewer skill is largely tacit, accumulated through extensive practice, and not easily acquired from short training.

EQUATOR / reporting-guideline frameworks make critique tractable.

The EQUATOR Network maintains the canonical inventory of reporting guidelines (CONSORT for RCTs, STROBE for observational studies, PRISMA for systematic reviews, ARRIVE for animal studies, STARD for diagnostic accuracy, CARE for case reports, SPIRIT for protocols). CONSORT was updated in 2025 (CONSORT 2025 statement, published simultaneously in several journals) [4, 12]. These guidelines turn “is this reported well?” into a checklist of specific items reviewers can flag [4]. Incomplete reporting against these guidelines is the problem they were written to address, which leaves room for actionable critique; this note does not cite a pooled adherence figure. Note the Cobo et al. result above: handing reviewers a checklist is not, on its own, enough to improve the manuscript [9].

Design principles for scriptorium’s reviewer-simulation

Translating the evidence into concrete design guidance:

Anchor critiques to specific manuscript passages and specific reporting items. Schroter et al. (2008) demonstrates that even motivated human reviewers miss most inserted errors. The remedy is not to demand the impossible (catch everything); it is to structure the search systematically. EQUATOR’s checklist-driven model is the design template: each persona walks through the relevant reporting-guideline items and flags absences or weaknesses, citing the manuscript passage. Because a checklist suggested to reviewers did not by itself improve quality in Cobo et al. (2007) [9], the checklist should drive concrete, passage-anchored findings rather than be appended as a generic list.
Optimize for the dimensions where review has demonstrable effect. Goodman et al. (1994) is the strongest evidence for where peer review improves manuscripts: limitations, generalizations, conclusions, reporting. Scriptorium’s simulated review should over-index on these dimensions because they are (a) demonstrably improvable by good critique and (b) tractable to assess from manuscript text alone. Cobo et al. (2007) adds that statistical review specifically produces measurable gains [9], which supports keeping a dedicated statistical lens.
Constructive form matters. The open-review trials (Walsh 2000, van Rooyen 1999) and Bornmann 2012 measured review quality, tone, and length — not whether authors acted on the critique — so they do not show directly that form changes author uptake. The practical pattern, supported by multiple guides rather than by trials: name the issue, point to the passage, explain why it matters, and (where possible) suggest a remedy. Treat the four-part pattern as well-motivated practice, not as an experimentally validated intervention.
Explicit non-findings are part of a good review. Because reviewers miss most errors (Schroter et al.), a review that says “I checked statistical analysis and have no concerns” carries more information than silence. Scriptorium personas should explicitly note which critique families they checked and found unproblematic — this is also the mechanism by which simulated review becomes auditable.
Volume is not quality. Bornmann, Weymuth & Daniel (2010, see common-critiques-taxonomy) found that a large number of negative comments in two specific areas — “Relevance of contribution” and “Design/Conception” — signalled that a rejected manuscript would not later appear in a high-impact journal, while the number of negative comments on writing, discussion of results, method/statistics, and literature had no significant influence. Scriptorium should not reward verbosity; the prioritization signal lives in critique family, not critique count.
Make the simulation inspectable like an open review. The open-review evidence (no loss of quality when reviewers are identified, and higher quality and courtesy in one trial) maps onto scriptorium’s “inspectable transformations” design principle directly. Every simulated critique is logged, attributable to a persona, anchored in text. This is open review as agentic protocol.

How this informs scriptorium

Build personas around the dimensions peer review provably improves (limitations, generalizability, conclusions, reporting quality), not the dimensions on which human reviewers under-perform (deep design-error detection). The simulation should be honest about what it can and cannot do.
Use reporting-guideline checklists (CONSORT, STROBE, ARRIVE etc.) as scaffolding for the methodological skeptic and statistical reviewer personas. The MANUSCRIPT_STATE.yaml could declare the relevant guideline; scriptorium then runs the checklist against the manuscript and reports item-level, passage-anchored findings (not the bare checklist).
Output format should follow the four-part open-review pattern: issue, passage reference, justification, suggested remedy. Critiques that don’t fit this shape should be flagged as low-quality by the skill itself.
Track explicit non-findings. Reviewer simulation should produce both “concerns” and “checked-no-concerns” sections; the latter is what distinguishes a structured review from a complaint generator.

Implementation priority for scriptorium

Verdict: Yes (shipped) — this note grounds the existing reviewer-simulation skill’s output contract. Scope: passage-anchored critiques; over-weighting limitations, generalizability, conclusions, and reporting; a dedicated statistical lens; explicit non-findings; no reward for critique volume. The 2026 refresh does not change the verdict; it narrows two supports (open review shows “no loss of quality” rather than “more constructive critique”, and checklists help only when turned into concrete findings). Required data: manuscript text, MANUSCRIPT_STATE.yaml (including a declared reporting guideline where applicable).

Open questions / weak evidence

Almost no RCT-quality evidence for AI-vs-no-AI authorial benefit. Thakkar et al. (2026, Nature Machine Intelligence; ICLR 2025, >20,000 reviews randomly selected to receive LLM feedback) is the closest analog — AI assistance to reviewers, not authors: 27% of reviewers who received feedback updated their reviews, and updated reviews were longer and rated more informative by blinded researchers [8]. The author-side experiment is open. Scriptorium’s own usage data could fill this gap.
Are checklist-grounded critiques actually more actionable than free-form ones? Plausible from the EQUATOR rationale, and the Cochrane review found “some evidence” for checklists [1], but the one randomized test of suggesting checklists to reviewers found no significant quality gain [9]. Whether agent-applied, item-by-item checklists behave differently is untested. A small in-house experiment within scriptorium’s user base would be feasible and informative.
The “constructive vs. destructive” literature is mostly normative. There are commentary papers and training guides; rigorous experimental comparisons of critique styles’ effect on author behavior are surprisingly thin. Sridhar’s “Constructive Peer Review Made Practical: A Guide to the EMPATHY Framework” (Journal of Marketing 89(3), 2025; doi:10.1177/00222429241312127) is one attempt to formalize this; it is a guide/framework paper rather than a controlled empirical study, so the framework should be treated as a structured proposal, not an empirically validated intervention.
Generalization across fields. Most strong evidence is from biomedicine (BMJ, Annals, Cochrane). Whether the same critique-quality patterns hold in CS, humanities, or qualitative social science is under-studied.
What does “useful critique” mean from the author’s perspective? “Author satisfaction” and “manuscript improvement” are different outcomes; the literature mostly measures one or the other, rarely both with the same design.

References

Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007(2): MR000016 (2007, multiple updates). DOI: 10.1002/14651858.MR000016.pub3. PMID: 17443635.
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 121(1): 11–21 (1994). DOI: 10.7326/0003-4819-121-1-199407010-00003. PMID: 8198342.
Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 101(10): 507–514 (2008). DOI: 10.1258/jrsm.2008.080062. PMID: 18840867.
EQUATOR Network. Reporting guidelines for health research. https://www.equator-network.org/reporting-guidelines/. CONSORT 2025: https://www.equator-network.org/reporting-guidelines/consort/.
Bornmann L, Wolf M, Daniel HD. Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics 91(3): 843–856 (2012). DOI: 10.1007/s11192-011-0569-5.
Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: randomized controlled trial. BMJ 328: 673 (2004). DOI: 10.1136/bmj.38023.700775.AE. PMID: 14996698.
White C. Little evidence for effectiveness of scientific peer review. BMJ 326(7383): 241 (2003). DOI: 10.1136/bmj.326.7383.241/a. PMID: 12560260. PMCID: PMC1125118. (News report of the 2003 Cochrane review; source of the “faith … rather than on facts” quotation.)
Thakkar N, Yuksekgonul M, Silberg J, Garg A, Peng N, Sha F, Yu R, Vondrick C, Zou J. A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence 8(3): 326–336 (2026). DOI: 10.1038/s42256-026-01188-x. Preprint: arXiv:2504.09737.
Cobo E, Selva-O’Callagham A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS ONE 2(3): e332 (2007). DOI: 10.1371/journal.pone.0000332. PMID: 17389922.
Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review: a randomised controlled trial. Br J Psychiatry 176: 47–51 (2000). DOI: 10.1192/bjp.176.1.47. PMID: 10789326.
van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 318(7175): 23–27 (1999). DOI: 10.1136/bmj.318.7175.23. PMID: 9872878.
Hopewell S, et al. (incl. Boutron I). CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ 389: e081123 (2025). DOI: 10.1136/bmj-2024-081123. PMID: 40228833.