/deep-research

Systematic Review Toolkit — Reference Guide

Purpose

Comprehensive reference for conducting systematic reviews and meta-analyses. Covers Cochrane methodology, PRISMA 2020 reporting, risk of bias instruments, heterogeneity interpretation, GRADE certainty framework, and protocol registration. Used by risk_of_bias_agent, meta_analysis_agent, bibliography_agent, and report_compiler_agent.

1. Cochrane Handbook v6.4 — Key Principles

The Cochrane Handbook for Systematic Reviews of Interventions (v6.4, 2023) is the gold standard reference for systematic review methodology.

Core Methodology Stages

Fundamental Principles

1. A priori protocol: Register the protocol before conducting the review (PROSPERO, OSF)
2. Comprehensive searching: Search multiple databases; do not rely on a single source
3. Dual independent processes: Two reviewers for screening, extraction, and risk of bias (at minimum for a subset)
4. Pre-specified methods: Analysis plan defined before seeing results
5. Transparent reporting: Document everything; another team should be able to replicate the review

2. PRISMA 2020 — Full 27-Item Checklist

Full Name: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Reference: Page et al. (2021). BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

Title and Abstract

Introduction

Methods

Results

Discussion

PRISMA 2020 Flow Diagram

 ┌─────────────────────────────────────────────────────┐
 │                 IDENTIFICATION                      │
 ├─────────────────────────────────────────────────────┤
 │ Records identified from databases (n = )            │
 │ Records identified from other sources (n = )        │
 └──────────────────────┬──────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────┐
 │ Records removed before screening:                   │
 │   Duplicate records (n = )                          │
 │   Records marked as ineligible by automation (n = ) │
 │   Records removed for other reasons (n = )          │
 └──────────────────────┬──────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────┐
 │                  SCREENING                          │
 ├─────────────────────────────────────────────────────┤
 │ Records screened (n = )                             │
 │ Records excluded (n = )                             │
 └──────────────────────┬──────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────┐
 │ Reports sought for retrieval (n = )                 │
 │ Reports not retrieved (n = )                        │
 └──────────────────────┬──────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────┐
 │ Reports assessed for eligibility (n = )             │
 │ Reports excluded, with reasons (n = )               │
 │   Reason 1 (n = )                                   │
 │   Reason 2 (n = )                                   │
 │   Reason 3 (n = )                                   │
 └──────────────────────┬──────────────────────────────┘
                        │
 ┌──────────────────────▼──────────────────────────────┐
 │                  INCLUDED                           │
 ├─────────────────────────────────────────────────────┤
 │ Studies included in review (n = )                   │
 │ Reports of included studies (n = )                  │
 │                                                     │
 │ Studies included in quantitative synthesis (n = )   │
 └─────────────────────────────────────────────────────┘

3. RoB 2 Instrument Summary

Full Name: Risk of Bias tool for randomized trials (version 2) Reference: Sterne et al. (2019). BMJ, 366, l4898. https://doi.org/10.1136/bmj.l4898

Domains

Judgment Scale

• Low risk of bias: The study is judged to be at low risk of bias for this domain
• Some concerns: The study raises some concerns about bias for this domain
• High risk of bias: The study is judged to be at high risk of bias for this domain

Overall Judgment Algorithm

• All domains Low → Overall Low
• Some Concerns in ≥ 1 domain, no High → Overall Some Concerns
• High in ≥ 1 domain → Overall High

4. ROBINS-I Summary

Full Name: Risk Of Bias In Non-randomized Studies of Interventions Reference: Sterne et al. (2016). BMJ, 355, i4919. https://doi.org/10.1136/bmj.i4919

Domains (7 domains spanning 3 time points)

Pre-intervention:

• D1: Bias due to confounding
• D2: Bias in selection of participants into the study

At intervention:

• D3: Bias in classification of interventions

Post-intervention:

• D4: Bias due to deviations from intended interventions
• D5: Bias due to missing data
• D6: Bias in measurement of outcomes
• D7: Bias in selection of the reported result

Judgment Scale

• Low risk: Comparable to a well-performed RCT
• Moderate risk: Sound for a non-randomized study but cannot be considered comparable to a well-performed RCT
• Serious risk: Some important problems
• Critical risk: Study is too problematic to provide useful evidence
• No information: Insufficient reporting

5. I² Interpretation Guide

Important caveats:

• Ranges overlap intentionally (per Cochrane Handbook 10.10.2)
• I² significance depends on: magnitude of effects, p-value from Q-test, and visual inspection of forest plot
• A high I² with all effects in the same direction is less concerning than moderate I² with effects crossing zero
• I² is influenced by precision of studies — many precise studies can yield high I² even with small absolute differences
• Always report the 95% CI for I² (which can be very wide with few studies)

6. GRADE Certainty of Evidence Framework

Full Name: Grading of Recommendations, Assessment, Development and Evaluations Reference: Guyatt et al. (2008). BMJ, 336, 924-926.

Starting Points

Factors That Lower Certainty (Rate Down)

Factors That Raise Certainty (Rate Up — Observational Studies Only)

Certainty Levels

7. Protocol Registration Guidance

When to Register

• Always for systematic reviews intended for publication
• Before starting the literature search
• Registration prevents outcome reporting bias and demonstrates a priori planning

Where to Register

Protocol Content (PRISMA-P 2015)

See templates/prisma_protocol_template.md for the complete protocol template.

Key sections:

1. Title, registration, authors, amendments
2. Rationale, objectives, PICOS eligibility criteria
3. Information sources, search strategy, study records management
4. Data extraction, risk of bias assessment, data synthesis plan
5. Meta-bias assessment, confidence in cumulative evidence

8. Software and Tools

Statistical Software for Meta-Analysis

Screening and Management Tools

Quick Decision Guide

Starting a systematic review?
│
├── 1. Register your protocol
│   └── PROSPERO (health) or OSF (any field)
│
├── 2. Write your protocol
│   └── Use PRISMA-P template → templates/prisma_protocol_template.md
│
├── 3. Search systematically
│   └── ≥ 2 databases, document everything, PRISMA flow
│
├── 4. Screen and select
│   └── Dual screening, predefined criteria
│
├── 5. Assess risk of bias
│   └── RCTs → RoB 2 | Non-randomized → ROBINS-I
│
├── 6. Synthesize evidence
│   ├── Quantitative data + comparable studies → Meta-analysis
│   └── Otherwise → Narrative synthesis (SWiM)
│
├── 7. Assess certainty
│   └── GRADE for each outcome
│
└── 8. Report
    └── PRISMA 2020 checklist → templates/prisma_report_template.md