Systematic Review Methodology

The gold standard for evidence synthesis in health research, and a roadmap for rigorous, transparent, and reproducible knowledge building.

Introduction

In the age of information overload, clinicians, policymakers, and researchers alike face a paradox: there is more data than ever before, yet making sense of it remains a daunting task. A single clinical question—Does a new antihypertensive drug reduce stroke risk in older adults?—might be answered by dozens of randomized trials, hundreds of observational studies, and countless conference abstracts. Without a disciplined approach, the sheer volume of evidence can lead to cherry‑picking, bias, and ultimately, decisions that jeopardize patient safety.

Systematic reviews were invented precisely to tame this chaos. By applying a pre‑specified, transparent protocol to locate, appraise, and synthesize all relevant studies, a systematic review transforms scattered findings into a single, defensible answer. The methodology has matured into a well‑defined science: the Cochrane Collaboration has published over 8,000 reviews since its inception, and PubMed now indexes more than 30,000 systematic reviews each year. Yet the rigor of a review depends not on the brand name but on how faithfully the author follows the step‑by‑step protocol—from question formulation to final reporting.

For the Apiary community, the parallel is striking. Bees thrive on collective intelligence: each forager shares information about flower quality, distance, and safety, allowing the colony to allocate resources efficiently. Likewise, a systematic review aggregates the “foraging” of individual studies, ensuring that the collective knowledge of the research community is used optimally. Moreover, emerging self‑governing AI agents—software that can autonomously screen citations, extract data, and flag bias—are beginning to act as the “worker bees” of evidence synthesis, promising faster, more reproducible reviews while preserving the human oversight essential for ethical decision‑making.

This pillar page walks you through every critical stage of systematic review methodology in health research. Whether you are drafting your first protocol, registering it on an international platform, or polishing the final manuscript for publication, the guidance below will help you produce a review that stands up to scrutiny, informs practice, and, in the spirit of Apiary, contributes to a healthier world for both humans and pollinators.

1. Defining a Systematic Review: Scope and Purpose

A systematic review is not a literature review, narrative essay, or expert opinion piece. It is a structured investigation that answers a specific, answerable question by identifying all relevant evidence, appraising its quality, and synthesizing the results in a reproducible manner. The key hallmarks are:

The purpose of a systematic review can be categorized into three broad families:

1. Effectiveness reviews – assess the impact of interventions (e.g., “Does vitamin D supplementation reduce falls in the elderly?”).
2. Diagnostic accuracy reviews – evaluate how well a test identifies a condition (e.g., “Accuracy of rapid antigen tests for COVID‑19”).
3. Prognostic reviews – synthesize evidence on outcomes after exposure (e.g., “Long‑term cardiovascular risk after gestational diabetes”).

Choosing the correct family determines which tools and statistical models you will use later. For instance, a diagnostic accuracy review typically employs the bivariate model for sensitivity and specificity, whereas an effectiveness review often uses a random‑effects meta‑analysis.

Real‑world example: A 2022 systematic review on bee pollen as a dietary supplement pooled data from 12 randomized controlled trials (RCTs) involving 1,845 participants. The review concluded that bee pollen modestly reduced LDL cholesterol (mean difference = ‑8.3 mg/dL, 95 % CI ‑13.2 to ‑3.4). By following the systematic methodology, the authors ensured that the conclusion was not driven by a single positive trial, thereby providing reliable evidence for clinicians considering nutraceutical options.

2. Formulating a Review Question

The foundation of any systematic review is a well‑crafted question. The most widely used framework in health research is PICO (Population, Intervention, Comparator, Outcome). For diagnostic and prognostic reviews, alternatives such as SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) or PECO (Population, Exposure, Comparator, Outcome) may be more appropriate.

2.1 The PICO Blueprint

A concise PICO question might read: In adults ≥ 50 years with hypertension, does daily supplementation with 5 g of dried bee pollen, compared with placebo, reduce systolic blood pressure after 12 weeks?

2.2 Refining the Scope

A common pitfall is an overly broad question that leads to an unmanageable number of studies. Conversely, an overly narrow question can result in an empty review. Strategies to balance scope include:

• Limiting the intervention (e.g., only standardized bee pollen preparations).
• Focusing on a specific outcome (e.g., systolic blood pressure rather than a composite cardiovascular endpoint).
• Applying a time‑frame filter (e.g., studies published after 2000 to capture modern formulations).

2.3 Registering the Question

Once the question is finalized, it should be locked in a publicly accessible protocol (see Section 3). Registering the question prevents “question‑drifting” and provides a timestamp that can be cited by journals and funders.

3. Protocol Development and Registration

A protocol is the blueprint of a systematic review. It details every methodological decision, ensuring that reviewers cannot unintentionally—or intentionally—alter the plan after seeing the data.

3.1 Core Elements of a Protocol

3.2 Choosing a Registration Platform

For health‑focused reviews, PROSPERO remains the gold standard. Registration requires a succinct abstract, detailed methods, and the anticipated completion date. Once approved, the record receives a unique identifier (e.g., CRD42020212345) that can be cited in the final manuscript.

Tip: Upload the full protocol as a supplementary file on OSF and link it from PROSPERO. This double‑layered approach maximizes transparency and safeguards against accidental loss of the protocol.

3.3 Protocol Peer Review

Some journals (e.g., Systematic Reviews and BMJ Open) offer protocol‑only peer review. Submitting your protocol for review before data extraction can catch methodological flaws early, saving months of work. Additionally, the peer‑reviewed protocol can be cited in the final article, demonstrating methodological rigor.

4. Literature Search Strategies

A systematic review’s credibility hinges on the comprehensiveness of its search. Inadequate searching is the leading cause of retraction or correction in systematic reviews.

4.1 Database Selection

A typical health systematic review searches at least three major databases (MEDLINE, Embase, CENTRAL) plus relevant trial registries.

4.2 Constructing Search Strings

A robust search string balances sensitivity (capturing all relevant records) and precision (excluding irrelevant hits). The process usually follows three steps:

1. Identify core concepts (e.g., “bee pollen”, “hypertension”).
2. Map synonyms and controlled vocabulary (MeSH, Emtree).
3. Combine with Boolean operators (AND, OR, NOT).

Example for bee pollen & hypertension

#1 "Bee Pollen"[Mesh] OR "bee pollen" OR melissae OR "melissae pollen"
#2 "Hypertension"[Mesh] OR hypertension OR "high blood pressure"
#3 "Randomized Controlled Trial"[Publication Type] OR "clinical trial"[pt] OR "RCT"
#4 #1 AND #2 AND #3

In Embase, replace [Mesh] with /exp for Emtree terms, and use :ti,ab,kw to limit to title/abstract/keyword fields.

4.3 Grey Literature & Hand‑Searching

Grey literature (conference abstracts, theses, government reports) can account for up to 30 % of all relevant evidence in certain fields. Strategies include:

• Searching OpenGrey, ProQuest Dissertations, and Google Scholar (first 200 results).
• Contacting study authors for unpublished data.
• Hand‑searching the tables of contents of key journals (e.g., Journal of Apicultural Research for bee‑related interventions).

4.4 Documenting the Search

Every search must be fully reproducible. Record the following for each database:

• Date of search (e.g., 12 Oct 2024).
• Exact search string (including line numbers).
• Limits applied (language, date range).

These details are typically presented in an appendix or as a supplementary table.

4.5 Automation and AI‑Assisted Screening

Self‑governing AI agents, such as ASReview and RobotReviewer, can prioritize citations based on relevance probabilities, reducing the manual workload by up to 70 % while maintaining > 95 % sensitivity. However, AI tools should complement—not replace—human judgment. In the Apiary ecosystem, AI agents can be trained on bee‑related datasets to improve the detection of niche terminology (e.g., “propolis”, “pollen load”).

5. Study Selection and Data Extraction

Once the search is complete, the next phase is screening (title/abstract then full‑text) and extracting the data needed for synthesis.

5.1 Dual Independent Screening

• Why two reviewers? A single reviewer may miss up to 15 % of eligible studies due to fatigue or bias. Dual screening reduces this error to < 5 %.
• Workflow: Use systematic review software (Covidence, Rayyan, or the open‑source RevMan). Import all citations, de‑duplicate, then assign each record to two blinded reviewers.
• Conflict resolution: A third reviewer adjudicates disagreements, or the two reviewers discuss until consensus is reached.

The outcome of the selection process is visualized in a PRISMA flow diagram, which records the number of records identified, screened, excluded, and finally included.

5.2 Extraction Forms

A structured extraction form captures all variables required for risk‑of‑bias assessment, effect‑size calculation, and subgroup analyses. Core fields include:

Pilot the form on three studies to ensure clarity and completeness.

5.3 Handling Missing Data

When essential data (e.g., standard deviations) are absent, follow a hierarchy:

1. Contact authors (up to three attempts).
2. Impute using methods recommended by the Cochrane Handbook (e.g., calculate SD from confidence intervals or p‑values).
3. Sensitivity analysis excluding imputed data to assess impact.

5.4 Data Management

Store extracted data in a version‑controlled repository (e.g., GitHub or OSF). Use CSV or JSON formats for easy import into statistical software (R, Stata, RevMan). Tag each commit with a meaningful message (e.g., “Added extraction for Smith 2021 RCT”).

6. Risk of Bias Assessment

Assessing methodological quality is not an optional add‑on; it informs the weight each study receives in the synthesis and underpins the GRADE rating of certainty.

6.1 Tools for Randomized Trials

• Cochrane Risk of Bias 2 (RoB 2) – evaluates five domains: randomization, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of reported results.
• ROBINS‑I – for non‑randomized studies, covering bias due to confounding, selection, classification of interventions, and more.

Both tools provide a judgment (low, some concerns, high risk) per domain and an overall rating.

6.2 Conducting the Assessment

1. Training: Reviewers should complete the official RoB 2 training module (≈ 2 hours).
2. Dual independent assessment: As with screening, two reviewers evaluate each study, documenting justifications for each judgment.
3. Consensus meeting: Resolve disagreements, and record the final decision in a risk‑of‑bias table.

6.3 Incorporating Bias into Synthesis

• Weighting: In a meta‑analysis, studies at high risk of bias can be down‑weighted using a quality effects model.
• Sensitivity analyses: Run the primary analysis with all studies, then repeat excluding high‑risk studies to see if conclusions change.

6.4 Visual Summaries

• Traffic‑light plots (green = low risk, yellow = some concerns, red = high risk) provide a quick visual snapshot.
• Risk‑of‑bias summary tables (available in RevMan) can be exported for the manuscript.

7. Data Synthesis

The synthesis stage translates extracted numbers into a coherent answer. The choice between quantitative (meta‑analysis) and qualitative (narrative) synthesis depends on heterogeneity, outcome type, and data availability.

7.1 When to Meta‑Analyse

• Homogeneous interventions (e.g., same bee‑pollen preparation).
• Comparable outcomes measured on the same scale (e.g., systolic blood pressure in mm Hg).
• Sufficient number of studies (≥ 2) to estimate between‑study variance.

If these conditions are not met, a narrative synthesis is more appropriate.

7.2 Calculating Effect Sizes

When studies use different scales (e.g., different blood pressure devices), compute the SMD (Cohen’s d).

7.3 Random‑Effects Model

Health interventions often exhibit clinical heterogeneity, so a random‑effects model is the default. The DerSimonian‑Laird estimator is widely used, but newer methods (e.g., Hartung‑Knapp–Sidik‑Jonkman) provide more accurate confidence