significance-positioning

Significance and positioning

Last updated: 2026-05-17

Synthesis

“Significance” in scientific writing is two distinct things that share a word. The first is rhetorical positioning: how the author frames why the work matters, why now, and to whom. The second is review-criterion significance: the specific question NIH or NSF reviewers are scored against, distinct from rigor, innovation, and approach. A manuscript or grant succeeds when its rhetorical positioning lands on the same point the applicable review criterion measures — and fails, often invisibly, when the two diverge.

The empirical literature on what reviewers actually weight is thinner than the volume of advice on the topic suggests. Tahamtan et al.’s 2016 review of citation predictors [1] is the cleanest meta-analysis we have, and points to a small number of consistently strong factors (paper quality, journal impact factor, number of authors, international collaboration, field visibility) with much of the rest of the literature being noisy. Empirical work on peer review (e.g., Wang et al. PNAS 2022 on novelty [2]) suggests novelty does predict acceptance, but reviewers rarely articulate it as such; it lives in implicit assessments.

For scriptorium, the practical implication is that the positioning section of a manuscript or grant (the abstract, the significance statement, the specific aims page, the closing paragraph of the introduction) is where the highest-leverage editorial work happens. reviewer-simulation (current) and a future positioning-audit skill should focus disproportionate attention here.

Evidence and frameworks

Day & Gastel: the classic procedural guide

Robert A. Day and Barbara Gastel’s How to Write and Publish a Scientific Paper, 8th edition (Greenwood / Cambridge University Press, 2016; ISBN-13 978-1-4408-4280-1; Cambridge ISBN 978-1-3166-4043-2) [3] is the longest- running general-purpose manual for scientific publishing. Day’s original 1979 edition has been continuously updated; Gastel joined as coauthor for recent editions and is now first author. The book is practical and procedural — title construction, abstract craft, author-instructions reading, response-to-reviewers — rather than theoretical. It is the default text in many graduate-level scientific writing courses outside biomedicine.

Day & Gastel’s framing of significance is conservative: state the problem, state what you did, state what is new, state why it matters, in roughly that order. The book does not propose a structural framework on the order of OCAR (narrative-frameworks) but it disciplines authors on the small craft moves (title weight-bearing, abstract structure, methods-as-recipe) that determine whether positioning lands.

Mewburn and the Thesis Whisperer tradition

Inger Mewburn, Director of Research Training at the Australian National University, has written extensively under the “Thesis Whisperer” brand (thesiswhisperer.com) on academic positioning, particularly for graduate students and early-career researchers. The relevant book-length treatment is How to Be an Academic: The Thesis Whisperer Reveals All (NewSouth Publishing, 2018; ISBN-13 978-1-7423-5073-3) [4].

Mewburn’s contribution is sociological more than rhetorical: she frames academic writing as a positioning practice within a community of practice, where the writer is making claims not just about the work but about their membership in a field. The Mewburn–Thomson 2013 paper [TODO verify exact citation; widely cited but I have not independently verified it in this session] on why academics blog is part of this lineage.

The practical takeaway for a writing system: significance claims read differently depending on the writer’s apparent positioning. A junior author claiming “first demonstration” carries less weight than the same claim from a senior author at a high-prestige institution. This is neither fair nor avoidable; the editorial response is to make the evidence for the claim do more work and rely less on author authority.

What predicts citation impact (Tahamtan et al. 2016)

Tahamtan I, Safipour Afshar A, Ahamdzadeh K. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics. 2016;107(3):1195–1225. DOI: 10.1007/s11192-016-1889-2. [1]

This review synthesized 198 empirical studies on what predicts citation counts. The three categories of factors:

1. Paper-related — quality, novelty/originality of subject, field characteristics, study type (review > original > letter), length, abstract characteristics, title characteristics.
2. Journal-related — impact factor, accessibility (open access), indexing, language (English dominant).
3. Author-related — number of authors, international collaboration, author seniority/h-index, institutional reputation.

The most consistent findings across studies: paper quality and journal impact factor dominate; international collaboration and number of authors contribute modestly; author demographics (age, gender, race) have weak and inconsistent effects.

For editorial work, the actionable factors are: title clarity and informativeness, abstract structure, choice of journal, and the inclusion of a clear methodological signal in the abstract. The non-actionable factors (institutional prestige, author seniority) bound the realistic ceiling of editorial improvement.

Empirical work on peer review and novelty

Wang J, Veugelers R, Stephan P. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy. 2017;46(8):1416–1436. DOI: 10.1016/j.respol.2017.06.006. [TODO verify exact DOI] [5]

More directly germane:

Lin J, Liang X, Frachtenberg E. Is novel research worth doing? Evidence from peer review at 49 journals. PNAS. 2022;119(33):e2118046119. DOI: 10.1073/pnas.2118046119. [2]

This study used reviewer-recommendation data from 49 journals across disciplines and showed (a) novelty positively predicted acceptance, (b) papers combining novel and conventional elements outperformed purely-novel ones, and (c) reviewers rarely used the word “novel” explicitly in comments — the preference operates implicitly.

The editorial implication: positioning that flags novelty and conventional grounding does better than positioning that flags novelty alone. The manuscript that opens with “Although the X approach is well established, we show…” is making both moves simultaneously.

NIH significance and the simplified review framework (2025)

As of January 25, 2025, NIH applications submitted under the Simplified Review Framework score against three factors rather than the five legacy criteria [6]:

• Factor 1: Importance of the Research (combining Significance and Innovation) — scored 1–9
• Factor 2: Rigor and Feasibility (Approach) — scored 1–9
• Factor 3: Expertise and Resources (Investigator, Environment) — rated for sufficiency, not scored

The Significance criterion as written: “Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?”

The reframing matters editorially. Under the old framework, an author could be strong on Significance and weak on Innovation (or vice versa) and get a mixed score. Under the new framework, both are bundled into Factor 1 — and a manuscript that nails one but not the other will score lower than one that ladders both into a single positioning claim.

NSF broader impacts criterion

NSF’s Merit Review Criterion 2 (“Broader Impacts”) was added in 1997 under Director Neal Lane and has been the subject of substantial empirical and philosophical analysis since [7, 8]. The criterion asks how the proposed work will advance societal goals beyond the immediate research outcome (education, workforce, public engagement, equity, infrastructure).

Holbrook (2005, 2009) [8] documents persistent community concerns: ambiguity about how broader impacts are weighted relative to intellectual merit; lack of evaluative consensus among reviewers; difficulty of projecting societal impact from basic research. A 2023 study (Marshall et al., Research Evaluation 32(2)) found measurable decline of broader impacts activity from proposal to project execution, suggesting many broader-impacts statements are aspirational rather than operational.

For editorial work, the implication: broader-impacts language should be specific (named partners, named programs, named metrics) rather than aspirational (“will broadly benefit underrepresented groups”). The specific form survives reviewer scrutiny better.

How this informs scriptorium

• reviewer-simulation (current). Should distinguish between the rhetorical-significance critique (“the framing doesn’t make me care”) and the criterion-significance critique (“under NIH simplified review, this doesn’t ladder Factor 1 well”). Knowing the target venue and the applicable review criteria changes what the simulator says.
• specific-aims-audit (planned, v0.4). NIH specific aims pages live or die on whether they articulate (i) a problem of acknowledged importance, (ii) a critical barrier, (iii) a tractable approach. The skill should test for each move explicitly. This is the most direct application of the Schimel OCAR + ABT diagnostic (narrative-frameworks) to the grant-writing domain.
• positioning-audit (proposed). A read-only critique skill that evaluates the abstract, significance statement, and introduction-closer for: (i) whether the “why now” is clearly stated, (ii) whether novelty is paired with conventional grounding, (iii) whether the named beneficiaries of the work are specific, not aspirational. Outputs a per-element assessment with span pointers.
• broader-impacts-audit (proposed, NSF-specific). Checks whether broader-impacts statements name specific partners, programs, and measurable activities — rather than aspirational claims. Could share infrastructure with positioning-audit.

The deeper design implication: positioning is the part of the manuscript where structured-output skills can have the highest leverage and the lowest semantic risk. The positioning sections are short, opinionated, and high-stakes. A scriptorium that focuses critique-skill density here produces visible improvements that authors can actually adopt.

Open questions / weak evidence

• The empirical literature on what reviewers weight is dominated by bibliometric proxies (citation counts) rather than direct measurement of reviewer decisions. The Lin et al. 2022 PNAS study [2] is unusual in using reviewer-level data; it is not yet replicated at scale.
• Mewburn’s positioning work is influential but pedagogical and qualitative rather than empirical. The claims about authority-effects on reviewer perception are plausible but not rigorously tested.
• NIH’s Simplified Review Framework is new (effective January 2025) and there is not yet enough scoring data to tell whether the factor consolidation has changed what reviewers prioritize.
• “Significance” as a criterion is famously vague. The empirical evidence that improving significance framing improves outcomes is indirect — largely from comparing accepted vs. rejected manuscripts after the fact, which is confounded with quality.

References

1. Tahamtan I, Safipour Afshar A, Ahamdzadeh K. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics. 2016;107(3):1195–1225. DOI: 10.1007/s11192-016-1889-2.
2. Lin J, Liang X, Frachtenberg E. Is novel research worth doing? Evidence from peer review at 49 journals. PNAS. 2022;119(33):e2118046119. DOI: 10.1073/pnas.2118046119.
3. Gastel B, Day RA. How to Write and Publish a Scientific Paper. 8th ed. Greenwood (Cambridge University Press); 2016. ISBN-13 978-1-4408-4280-1; Cambridge paperback ISBN-13 978-1-3166-4043-2.
4. Mewburn I. How to Be an Academic: The Thesis Whisperer Reveals All. NewSouth Publishing; 2018. ISBN-13 978-1-7423-5073-3.
5. Wang J, Veugelers R, Stephan P. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy. 2017;46(8):1416–1436. [TODO verify DOI 10.1016/j.respol.2017.06.006]
6. National Institutes of Health. Simplified Peer Review Framework. NOT-OD-24-010 and follow-on policy notices. https://grants.nih.gov/policy-and-compliance/policy-topics/peer-review/simplifying-review (accessed 2026-05-17; effective for applications submitted on or after January 25, 2025).
7. National Science Foundation. Merit Review Broader Impacts Criterion: Representative Activities. NSF; ongoing. https://www.nsf.gov/funding/learn/broader-impacts
8. Holbrook JB. Assessing the science–society relation: The case of the US National Science Foundation’s second merit review criterion. Technology in Society. 2005;27(4):437–451. (See also Holbrook, J.B., “The use and abuse of the peer review process for funding decisions,” Social Epistemology 2009;23(3–4):219–237.) [TODO verify exact 2005 DOI]