Review Quality Thinking Framework
A cognitive framework for producing high-quality reviews. Teaches how to think about paper quality, not just which judgement label to emit.
The Three Lenses
Every paper should be evaluated through three lenses simultaneously:
Lens 1: Internal Validity — "Does the evidence support the claims?"
Ask in order:
- What is the central claim?
- What evidence is presented?
- Is there a logical chain from evidence to claim? (Warrant)
- Are there alternative explanations the authors didn't consider?
- Would removing any single piece of evidence collapse the argument?
If #5 is yes: The argument depends on a single linchpin. Flag it — the paper's contribution is only as strong as that one piece of evidence.
Lens 2: External Validity — "Does this matter beyond this study?"
Ask in order:
- Who is the population of interest?
- Does the sample represent that population?
- Are the conditions replicable?
- Would the findings hold in a different context/culture/time?
- What are the boundary conditions the authors don't mention?
Judgment heuristic: Check the manuscript's claimed scope of generalization against its sampling frame — a criterion-only comparison, never an author-population base rate (#574 B1). If the sample is from one university in one country, findings cannot claim to apply "generally" without qualification; if the claims already match the evidence's scope, there is no finding here.
Lens 3: Contribution — "So what?"
Ask in order:
- What did we know before this paper?
- What do we know after?
- Is the delta meaningful (not just statistically significant)?
- Who benefits from knowing this?
- What new questions does this open?
Judgment heuristic: If you can't articulate the delta in one sentence, either the contribution is weak or the paper hasn't communicated it clearly. Both are review-worthy observations.
Common Reviewer Traps
| Trap | Description | How to Avoid |
|---|---|---|
| Methodological tunnel vision | Only critiquing methods, ignoring whether the question matters | Start with Lens 3 (contribution) before Lens 1 |
| Novelty bias | Penalizing replication or incremental work | Replication IS valuable; evaluate on execution quality |
| Expertise projection | Expecting the paper to use your preferred method | Evaluate the chosen method on its own terms |
| Positivity-severity oscillation | Being too nice in comments, too harsh in the final judgement | Commit the criterion, evidence, and categorical judgement before polishing the comments |
| Missing forest for trees | Listing 20 minor issues, missing the one fatal flaw | Always state the single most important issue first |
Calibration Questions (ask after drafting your review)
- If this paper were published as-is, would it mislead readers? (If yes → Major Revision or Reject)
- Could the authors reasonably address my concerns in one revision cycle? (If no → Reject)
- Am I being harder on this paper than I would be on my own work? (Calibration check)
- Did I acknowledge every genuine strength I found — and manufacture none? (Evidence-driven balance check, #574 A1/B1)
- Would my review help the authors improve, even if the paper is rejected? (Constructiveness check)