Citation accuracy: evidence base

2026-05-17

Last updated: 2026-10-20

Synthesis

The empirical literature on citation accuracy converges on an uncomfortable finding: roughly one in four to one in five citations in biomedical and adjacent literatures contains some form of inaccuracy, and roughly one in ten contains an inaccuracy serious enough to mislead a careful reader about what the cited paper says. This is not a “new” problem traceable to LLMs; it predates them by four decades. The earliest systematic accounting was published in BMJ in 1985 (de Lacey et al. 1985), and every replication since — across specialties, decades, and review methodologies — has landed in the same neighborhood.

Two patterns matter most for any system that audits citations. First, errors are not evenly distributed: review articles and frequently cited “canonical” papers attract disproportionate inaccuracy, and a substantial minority of errors propagate as citation chains in which authors copy a wrong claim from a previous author who copied it from theirs. Second, errors split between quotation errors (the cited paper does not support the statement, or supports it only partially) and citation errors (bibliographic data is incomplete or wrong). Quotation errors are the load-bearing problem; bibliographic errors are mostly cosmetic.

Implication for citation-audit: the skill should be designed around the quotation-support question — does the cited work actually substantiate the in-text claim? — rather than around bibliographic metadata, where the marginal value of an automated check is small and existing tools (CrossRef, reference managers) are already strong.

Evidence

de Lacey et al. (1985) examined references in six medical journals published in January 1984. Quotation accuracy: the original author was misquoted in 15% of all references, and “most of the errors would have misled readers.” Citation accuracy: errors occurred in 24% of references, of which 8% were major (preventing identification of the source). This is the foundational study; subsequent work largely confirms its numbers.

The Wager and Middleton (2008) Cochrane methodology review (Wager & Middleton 2008, Technical editing of research reports in biomedical journals) located 66 surveys of reference accuracy, which together checked over 27,000 references in over 100 journals. The median citation-error rate per journal was 38% (range 4–67%), and the median quotation-error rate per journal was 20% (range 0–50%); pooled, 761 of 3,836 references (20%) assessed for quotation accuracy were quoted inaccurately. More intensive editorial processes were associated with fewer reference errors, but the evidence for that is weak: only three of the 32 technical-editing studies in the review were randomised controlled trials, many of the others compared before and after, and the review describes most of them as small and probably under-powered. Most of the reference-accuracy surveys were single-time-point “snapshots” of one or a few journals.

Jergas and Baethge (2015) — systematic review and meta-analysis of 28 studies (1985–2013) covering 7,321 references. Pooled prevalence: major quotation errors 11.9% (95% CI 8.4–16.6); minor errors 11.5% (95% CI 8.3–15.7); total quotation errors 25.4% (95% CI 19.5–32.4). Their conclusion: readers “should be aware of the fact that quotation errors are common,” and “even the lowest estimate of total quotation errors was considerable (6.7%).”

Pavlovic et al. (2021) re-examined frequently cited biomedical papers by going back to the original first authors to verify what their own papers actually said. Feasibility study (1,540 articles, 2,526 citations of 14 papers): 7.2% of individual citations inaccurate; 11.1% of articles contained at least one inaccuracy. Verification study (2,995 articles, 4,912 citations of 13 papers): 10.3% citations inaccurate; 15.0% of articles affected.

Critical sub-findings from Pavlovic et al. (2021):

• Citation of nonexistent findings was the largest single error category (38.4% of inaccuracies) — the cited paper did not contain the claim attributed to it at all.
• Inaccurately cited numerical data (16.6%) and inaccurate interpretation (15.4%) were the next two categories.
• Citation chains accounted for ~19–24% of inaccuracies (24% in the feasibility study, 19.3% in the verification set) — errors copied from previous citing articles rather than introduced fresh.
• Review articles were more likely than primary research articles to contain inaccuracies, and inaccuracy rose with time since the cited paper’s publication.

Specialty-specific evidence — Sauder et al. (2022) (Sauder et al. 2022, The American Surgeon) evaluated 5,973 citations from highly cited studies in ten top-ranked surgical journals (2015–2020) and found 15.2% inaccurate, broadly consistent with the general biomedical figures. It found no statistically significant difference in inaccuracy by year or by study level of evidence (P = .21); differences were by journal. The article carries a January 2025 publisher expression of concern (doi:10.1177/00031348241305412) that was issued across a large batch of American Surgeon articles, so its figures should be treated as corroborating rather than load-bearing.

How this informs scriptorium

The evidence base shapes citation-audit in three concrete ways.

1. Audit the quotation, not just the citation. Bibliographic errors are a long-solved problem in modern reference-manager workflows; quotation errors are not. The skill’s primary responsibility is to ask, for each in-text citation, whether the referenced work supports the specific claim being made. Output sections should make this question explicit (“Claim supported?” “Claim partially supported?” “Claim not located in source?”).

2. Treat review citations differently from primary citations. The Pavlovic et al. (2021) finding that review articles carry higher inaccuracy rates, combined with the “review-citing-review” cascade documented in [[citation-overreach-research]], means the auditor should flag when a mechanistic or quantitative claim is supported only by a review citation. The flag is not “wrong” — it is “verify with primary source.” This recommendation should never invent the primary source; see [[hallucination-in-llm-citations]].

3. Surface citation chains. When two or more papers cite the same source for the same claim using nearly identical phrasing, this is a hallmark of an unverified citation chain. The skill can detect such patterns when multiple papers in a corpus are audited together, and warn — without claiming to have verified the underlying source.

In MANUSCRIPT_STATE.yaml, the preserve_citations: true constraint prevents transformation skills (compression, argumentative-flow) from silently dropping or replacing references — which would otherwise introduce the citation errors documented above as a side effect of helpful-looking edits.

Implementation priority for scriptorium

Verdict: Yes (v0.1).

Skill: citation-audit (critique phase). This note supplies the prevalence baselines the skill’s framing rests on: a ~20% median quotation-error rate per journal (Wager & Middleton 2008), a pooled 25.4% total quotation-error rate (Jergas & Baethge 2015), and the review-article and citation-chain risk factors (Pavlovic et al. 2021). Required data: the manuscript, its bibliography, and — for anything stronger than a metadata-only assessment — the cited abstracts or full texts. The 2026-10 refresh did not change the verdict; it corrected two figures (the Cochrane review’s denominators and the surgical study’s level-of-evidence finding), neither of which the skill depends on.

Open questions / weak evidence

• Most accuracy studies are biomedical. Engineering, physics, and the humanities are less well-characterized. The numbers above should not be extrapolated to those literatures without caveat.
• “Major” vs “minor” error categorization is judgment-dependent. Jergas and Baethge (2015) acknowledges heterogeneity; pooled estimates carry wide confidence intervals.
• The fraction of errors that change scientific conclusions (vs. merely misattribute who said what) is not well quantified. Greenberg’s distortion analysis in [[citation-overreach-research]] is the closest existing work and is qualitative.
• “Bibliographic errors are mostly solved” holds for journal articles with DOIs. Reference managers still mishandle other output types — a 2025 study found most frequently used reference managers do not adequately support dataset citation (see [[reference-managers]]).

Build note

This document is rendered from citation-accuracy-evidence.qmd via Quarto with a quartobot pre-render step. Cite keys in prose use persistent identifiers directly (@pmid:, @doi:); quartobot resolves them through manubot into references.json before Quarto’s citeproc renders the final citations and the References section. No hand-curated .bib file is needed — the cite key is the source of truth, and the bibliography is recomputed on every build. This is the first knowledge-layer document on the quartobot pipeline; the preprocess script (docs/scripts/preprocess.py) handles the resolve-then-render dance.

References

Jergas, Hannah, and Christopher Baethge. 2015. “Quotation Accuracy in Medical Journal Articles-a Systematic Review and Meta-Analysis.” PeerJ 3 (October): e1364. https://doi.org/10.7717/peerj.1364.

Lacey, G de, C Record, and J Wade. 1985. “How Accurate Are Quotations and References in Medical Journals?” British Medical Journal (Clinical Research Ed.) 291 (6499): 884–86. https://doi.org/10.1136/bmj.291.6499.884.

Pavlovic, Vedrana, Tracey Weissgerber, Dejana Stanisavljevic, et al. 2021. “How Accurate Are Citations of Frequently Cited Papers in Biomedical Literature?” Clinical Science (London, England : 1979) 135 (5): 671–81. https://doi.org/10.1042/cs20201573.

Sauder, Matthew, Kevin Newsome, Israel Zagales, et al. 2022. “Evaluation of Citation Inaccuracies in Surgical Literature by Journal Type, Study Design, and Level of Evidence: Towards Safeguarding the Peer-Review Process.” The American Surgeon 88 (7): 1590–600. https://doi.org/10.1177/00031348211067993.

Wager, Elizabeth, and Philippa Middleton. 2008. “Technical Editing of Research Reports in Biomedical Journals.” Cochrane Database of Systematic Reviews 2010 (1). https://doi.org/10.1002/14651858.mr000002.pub3.