Skip to content
Scriptorium Scriptorium

Reviewer Archetypes: What the Evidence Shows

Last updated: 2026-10-20

Synthesis

Section titled “Synthesis”

Empirical research on peer review converges on an uncomfortable result: when two competent reviewers read the same manuscript, they agree only modestly on its merit. The largest meta-analysis to date (Bornmann, Mutz & Daniel 2010, 48 studies, ~19,443 manuscripts) reports a mean inter-rater reliability of Cohen’s κ ≈ 0.17 and ICC/r² ≈ 0.34 — i.e., reviewer agreement is “low” by any standard psychometric benchmark [1]. This is not a flaw to be reformed away; it is a stable empirical regularity across decades, fields, and journal types. Different reviewers attend to different aspects of the same work. That fact, more than any other, justifies a multi-persona reviewer simulation: a single LLM “reviewer” trained to mimic the field’s median is by construction less useful than a panel that deliberately diversifies the attentional lens.

The bias literature is more contested than its popular framing suggests. Lee, Sugimoto, Zhang & Cronin (2013) systematically reviewed the field and concluded that “a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias” [2]. The clearest experimentally-demonstrated bias is prestige bias under single-blind review (Tomkins, Zhang & Heavlin 2017 PNAS, a controlled experiment at the 2017 WSDM conference) — single-blind reviewers were significantly more likely to recommend acceptance for papers from famous authors, top universities, and top companies (estimated odds multipliers 1.63, 1.58, and 2.10 respectively), and during bidding they bid for 22% fewer papers and preferentially bid for papers from top universities and companies [3]. Gender bias, by contrast, has not held up under the largest test: Squazzoni et al. (2021, Science Advances, 145 journals, ~1.7M authors, ~740K reviewers) found no systematic penalty against women authors, and in some segments a slight pro-women effect [4].

The folkloric “Reviewer 2” — the rude, smug, pet-theory-obsessed referee — has weaker empirical support than the meme suggests. The one direct test we have (Peterson 2020, satirical title notwithstanding; reviewer database of the journal Political Behavior) found that reviewers labeled “2” by the editorial system are not systematically harsher than “1”; it was Reviewer 3 who was most likely to rate a manuscript more than one category below the other reviewers [5]. The trope captures something real (unprofessional reviews exist and are common — Silbiger & Stubler’s survey of 1,106 STEM researchers, published in PeerJ in 2019, found 58% had received at least one unprofessional review [6]) but assigns it to the wrong position number. For a simulation system, the takeaway is that the “harsh skeptical reviewer” archetype is empirically defensible as a behavior, not as a numbered role.

Evidence

Section titled “Evidence”

  • Inter-rater reliability is low and stable. Bornmann, Mutz & Daniel (2010), the canonical meta-analysis, pooled 70 reliability coefficients from 48 studies of journal peer review: mean Cohen’s κ = 0.17, mean ICC/r² = 0.34. The authors conclude IRR is “quite limited and needs improvement.” Earlier work reached the same conclusion: Cicchetti’s (1991) cross-disciplinary review in Behavioral and Brain Sciences focused on “the consistently low levels of reliability that have been reported” for manuscript and grant review [1, 7].

  • Bias literature is more nuanced than slogans. Lee et al. (2013) is the most cited critical review. Many bias claims rely on observational data with weak controls; the experimental record is thinner. The strongest experimental finding is Tomkins et al. (2017): at WSDM 2017, each submission was scored by two single-blind and two double-blind program-committee members drawn from disjoint pools, and single-blind reviewers were significantly more likely than their double-blind counterparts to recommend acceptance for papers from famous authors, top universities, and top companies. Single-blind reviewers also bid for 22% fewer papers and preferentially bid for papers from top universities and companies [3]. An often-cited earlier result on double-blinding, Budden et al. (2008), is observational and concerns gender rather than prestige: it reported an increase in female first-authored papers at Behavioral Ecology after the journal adopted double-blind review in 2001; the finding has been contested [8].

  • Gender bias does not appear at scale. Squazzoni et al. (2021) is the largest study of its kind: 145 journals across biomedicine, life sciences, physical sciences, and social sciences/humanities (data shared by several large publishers under the PEERE data-sharing protocol), ~1.7M authors, ~740K referees, full submission-to-decision data. Result: “manuscripts written by women as solo authors or coauthored by women were treated even more favorably by referees and editors” [4]. The signal in smaller studies appears to wash out at scale, though field-specific effects (notably in some humanities/social sciences) remain plausible. A contrasting single-journal analysis is noted in the next bullet but one.

  • “Reviewer 2” is folklore that captures real behavior at the wrong address. The trope is academic culture (memes, the “Reviewer 2 Must Be Stopped!” Facebook group). Silbiger & Stubler’s survey reported that 58% of 1,106 STEM respondents had received an unprofessional review; respondents across gender and race/ethnicity groups received such comments equally, but traditionally underrepresented groups were most likely to report negative effects on perceived aptitude, productivity, and career advancement [6]. The empirical test of the position-number claim (Peterson 2020: Political Behavior reviewer database, 1,323 manuscripts over four years) found no evidence that Reviewer 2 was more negative or more out of line with the other reviewers; Reviewer 3 was significantly more likely to be the negative outlier in that dataset [5]. So: harshness exists, position numbering doesn’t predict it.

  • Reviewer–author demographics can interact. Murray et al. (2019, bioRxiv preprint; 23,876 initial and 7,192 full submissions to eLife, 2012–2017) found author–gatekeeper homophily: acceptance rates were higher when reviewers/editors shared the author’s gender or country, and the acceptance gap favouring male last authors (3.5 percentage points overall) was largest when the reviewer team was all-male [9]. This single-journal result sits in some tension with the Squazzoni et al. null at scale [4]. (An earlier version of this note also claimed that early-career reviewers write longer reports and senior reviewers are more decision-oriented; that claim could not be traced to Squazzoni et al. or Murray et al. and has been removed.)

How this informs scriptorium

Section titled “How this informs scriptorium”

The four-persona design in scriptorium’s reviewer-simulation skill is grounded in this evidence in three ways:

  1. Diversity-of-attention is the point, not consensus. Because real reviewers agree at κ ≈ 0.17, a simulation that produces four convergent reviews is less faithful to the literature than one that produces four divergent ones. The methodological skeptic, domain expert, translational reviewer, and statistical reviewer should each surface critiques the others miss. Convergence on a critique becomes a stronger signal than agreement on praise.

  2. Personas are attentional lenses, not numbered slots. Avoid labels like “Reviewer 1 / Reviewer 2 / Reviewer 3” — that framing imports folklore the data don’t support. Label personas by the lens they apply (methodological / domain / translational / statistical). This also avoids inheriting the asymmetric “harshness” connotations of numbered roles.

  3. Acknowledge prestige and confirmation bias in design. Because single-blind prestige bias is the clearest experimentally demonstrated reviewer bias [3], reviewer-simulation should not be primed with author identity, institutional affiliation, or venue prestige. The skill should operate on the manuscript text and MANUSCRIPT_STATE.yaml claims/limitations only. This is a defensible default the human-reviewer literature actively endorses (Tomkins et al.’s implicit recommendation was double-blinding).

Implementation priority for scriptorium

Section titled “Implementation priority for scriptorium”

Verdict: Yes (shipped) — this note grounds the existing reviewer-simulation skill. Scope: the four-lens design, the lens-not-numbered-slot labelling, and the rule that author identity, affiliation, and venue prestige are not fed to the lenses. The 2026 evidence that LLM reviewers overlap with each other far more than human reviewers do [10] strengthens, rather than changes, the case for deliberately diversified lenses; it does not by itself require a new skill. Required data: manuscript text and MANUSCRIPT_STATE.yaml only.

Open questions / weak evidence

Section titled “Open questions / weak evidence”
  • Do LLM-simulated reviewers reproduce human reviewer disagreement patterns? Liang et al. (2024) report ~30% point-level overlap between GPT-4 and human reviewers, comparable to the ~28% overlap between two human reviewers on the same papers [see ai-peer-review-research.md]. Newer evidence points the other way on diversity: in an expert-annotation study of 2,960 criticisms on 82 Nature-family papers, Kim et al. (2026, arXiv preprint, “work in progress”) found that AI reviewers overlapped with each other far more than human reviewers did (21% vs. 3% for cross-reviewer pairs), even though the best AI reviewer’s criticisms scored higher on a composite of correctness, significance, and sufficiency of evidence than each paper’s top-rated human reviewer [10]. Whether persona prompting recovers human-like diversity is unresolved.
  • Field-specificity of bias findings. Squazzoni et al. (2021) draws on large commercial publishers’ journals, and social sciences and humanities make up only 16 of the 145 journals; small-society journals are underrepresented. Gender-bias null findings may not generalize.
  • Persona orthogonality. No empirical work directly tests whether the four personas scriptorium uses are independent in critique space, or whether (e.g.) “methodological skeptic” and “statistical reviewer” mostly produce the same critiques in practice. This is a first-class empirical question for scriptorium itself to investigate from its own logs.
  • Whether disagreement is signal or noise. Bornmann et al. (2010) treat low IRR as a defect. Others (e.g., Lee 2012, “A Kuhnian critique of psychometric research on peer review”) argue that psychometric framing “overlooks different forms of normatively appropriate disagreement among reviewers” [11]. Scriptorium’s design implicitly takes the latter view.

References

Section titled “References”
  1. Bornmann L, Mutz R, Daniel HD. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5(12): e14331 (2010). DOI: 10.1371/journal.pone.0014331. PMID: 21179459.
  2. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inf Sci Tech 64(1): 2–17 (2013). DOI: 10.1002/asi.22784.
  3. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. PNAS 114(48): 12708–12713 (2017). DOI: 10.1073/pnas.1707323114. PMID: 29138317.
  4. Squazzoni F, Bravo G, Farjam M, Marusic A, Mehmani B, Willis M, Birukou A, Dondio P, Grimaldo F. Peer review and gender bias: a study on 145 scholarly journals. Sci Adv 7(2): eabd0299 (2021). DOI: 10.1126/sciadv.abd0299. PMID: 33523967.
  5. Peterson DAM. Dear Reviewer 2: Go F’ Yourself. Social Science Quarterly 101(4): 1648–1652 (2020). DOI: 10.1111/ssqu.12824.
  6. Silbiger NJ, Stubler AD. Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ 7: e8247 (2019). DOI: 10.7717/peerj.8247. PMID: 31844596.
  7. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behavioral and Brain Sciences 14(1): 119–135 (1991). DOI: 10.1017/S0140525X00065675.
  8. Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R, Lortie CJ. Double-blind review favours increased representation of female authors. Trends Ecol Evol 23(1): 4–6 (2008). DOI: 10.1016/j.tree.2007.07.008. PMID: 17963996.
  9. Murray D, Siler K, Larivière V, Chan WM, Collings AM, Raymond J, Sugimoto CR. Author-reviewer homophily in peer review. bioRxiv preprint (2019). DOI: 10.1101/400515.
  10. Kim S, Yoon D, Gashteovski K, et al. (incl. Neubig G). On the limits and opportunities of AI reviewers: reviewing the reviews of Nature-family papers with 45 expert scientists. arXiv:2605.20668 (2026, preprint). https://arxiv.org/abs/2605.20668.
  11. Lee CJ. A Kuhnian critique of psychometric research on peer review. Philosophy of Science 79(5): 859–870 (2012). DOI: 10.1086/667841.

See also: common-critiques-taxonomy, critique-quality-evidence, ai-peer-review-research.