“Studies show” — how to check whether it’s one study or a systematic review

Blog · science and source verification 1 STUDY SYSTEMATIC REVIEW
approx. 16 min read Category: Blog Updated: September 2026

The AI Blog team — content verified against Cochrane guidelines, the PRISMA standard and documented cases from The Lancet, Vaccine and PNAS (full list of sources in the text), current as of September 2026.

Short answer: Open the source scientific article (not just the popular headline) and check three things: the publication type given next to the title (Randomized Controlled Trial, Systematic Review, Meta-Analysis, Case Report), the number of participants and studies included in the analysis, and whether there’s a PRISMA diagram or a registration number in the PROSPERO database. If the text mentions only one experiment on a few dozen people — it’s a single study, not a systematic review, no matter how firmly the headline puts it.

Why it matters at all

In May 2015, journalist John Bohannon published a “study” that supposedly proved that eating chocolate speeds up weight loss. The experiment was real — but deliberately designed to produce a statistically significant yet worthless result: 16 people, a dozen or so measured variables, no correction for multiple comparisons. The story made the front pages of newspapers in more than a dozen countries as “studies show chocolate helps you lose weight.” Bohannon himself later described the whole hoax to show how easily a single, weak study turns into a media fact.

It’s not an isolated case. The most notorious is the story of Andrew Wakefield, who in 1998 published an article in the prestigious The Lancet suggesting a link between the MMR vaccine and autism — based on just 12 children. The article was retracted only in 2010, when the falsified data came to light and the author lost his medical license. Meanwhile, in 2014 a meta-analysis by Taylor and colleagues in the journal Vaccine, covering data from more than 1.2 million children from many cohort and case-control studies, found no link at all. A single study and a systematic review of dozens of studies can lead to completely different conclusions — and a media headline rarely tells you which one you’re dealing with.

📊 The effect is measurable: when hundreds of news articles about medical research were analyzed in 2020, articles based on a single, small study were much more likely to draw stronger cause-and-effect conclusions than the source material justified.

What the difference is

A single study answers a question once, under one set of conditions, in one group of participants. A systematic review doesn’t run a new experiment — it gathers all the available studies on a topic, assesses their quality according to clear criteria and synthesizes the results. A meta-analysis goes one step further: it combines numerical data from many studies into a single, statistically weighted result. That’s why in the hierarchy of scientific evidence, systematic reviews and meta-analyses usually rank highest, above single randomized trials, observational studies or case reports.

📄

A single study

One team, one group of participants, one moment in time. The result may be due to chance, depend on the specific sample or on a methodological error nobody has verified yet.

🗂️

A systematic review

Dozens or hundreds of studies, searched in several databases according to a pre-registered protocol, assessed for risk of bias and synthesized into one conclusion.

That doesn’t mean a single study is worthless — every systematic review is made up of single studies, after all. The problem arises when one preliminary, often small study is presented in the media as if it were final, established scientific proof.

15 ways to check

The list below consists of concrete, technical tricks — from one click in PubMed to analyzing a chart in a publication — that let you establish in a few minutes what kind of scientific evidence you’re really dealing with.

Find and read the original publication, not just the news article

Media coverage almost always links to (or at least mentions) the source publication or the university/institute that released it. Enter the authors’ names and the study’s title into Google Scholar or PubMed — it’s the first and most important step, because all the other tips require access to the original.

Check the “Publication type” right next to the title in PubMed

Every PubMed record has a publication type label: Randomized Controlled Trial, Observational Study, Case Reports, Review, Systematic Review or Meta-Analysis. It’s one line of text that settles the question immediately — you don’t need to read the whole thing.

Use the “Article Type” filter in the search results panel

On the left side of PubMed results (and similarly in Europe PMC) you can tick the “Systematic Review” or “Meta-Analysis” filter to narrow the list immediately to secondary research on the topic — useful when you want to check whether such a review exists at all.

Count the number of participants (n=)

A single clinical study usually involves anywhere from a dozen to a few hundred people. A systematic review adds up data from many studies — think thousands or millions, as in the meta-analysis on vaccines and autism mentioned above (more than 1.2 million children). A small sample doesn’t disqualify a study, but it limits how broad the conclusions drawn from it can be.

Look for a PRISMA diagram

Reliable systematic reviews are reported according to the PRISMA standard and include a characteristic flow chart: how many publications were found, how many were excluded at each stage and how many were ultimately included in the analysis. The absence of such a diagram in an article that calls itself a “review” is a warning sign.

Check whether the protocol is registered in PROSPERO

Credible systematic reviews (especially medical ones) are registered in advance in the international PROSPERO database before the authors start collecting data — the registration number usually appears in the abstract or the methods section. This guards against picking results to fit a predetermined thesis.

Read the “Methods” section — how many databases were searched

A solid review describes searching at least several independent databases (e.g., PubMed, Embase, Cochrane Library, Web of Science, Scopus) over a specific period. If the methods section describes only one experiment on one group of participants, it’s not a systematic review, whatever the title says.

Find the forest plot

The characteristic chart with a row of horizontal lines and a diamond symbol at the bottom is a “forest plot” — the visual signature of a meta-analysis. Each line is one source study, and their combined result is shown at the very bottom. If a publication has such a chart, it almost certainly combines data from many studies.

Check the I² heterogeneity statistic

In meta-analyses it’s worth glancing at the I² statistic, which shows how much the results of individual studies differ. Values of 0–40% usually mean low heterogeneity, while 75–100% means you should treat the combined conclusion with great caution — even if it’s formally a “systematic review.”

Note whether it’s a study on humans, animals or cells

Headlines like “studies show ingredient X cures cancer” very often describe results in mice or in cell cultures in the lab (in vitro), not in humans. It’s still valuable science, but a completely different level of evidence from a clinical trial — check it in the methods section, not in the headline.

Tell a “case report” apart from a population study

A case report describes one or a few patients — the lowest level of evidence in the evidence-based medicine hierarchy, yet it’s sometimes cited in the media as a full-fledged “study.” The words “case report” or “case series” in the title immediately set the right expectations.

Check whether it’s a preprint or a peer-reviewed article

Preprints published on servers such as medRxiv or bioRxiv haven’t yet gone through peer review — they may change or be rejected. These platforms explicitly label texts as preprints; it’s worth checking before you treat a result as final.

Check the journal and the study in the Retraction Watch Database

Retraction Watch runs a publicly available database of retracted scientific publications. If a famous “study” from years ago is still circulating on social media, it’s worth checking in this database whether it has already been officially retracted — as happened to Wakefield’s article in 2010.

Check whether there’s already a Cochrane review on the topic

The Cochrane Library collects some of the most rigorous systematic reviews in medicine, updated regularly. Before you treat a single new news report as a breakthrough, check Cochrane’s search to see whether the topic has already been studied many times and systematically — the answer is often more measured than the headline.

Compare the language: “suggests” or “proves”

Authors of single, preliminary studies usually write cautiously: “the results suggest,” “may indicate a link,” “further research is needed.” When a popular article turns that into a firm “scientists have proven” or “research clearly shows,” it’s most often journalistic over-interpretation, not the authors’ own conclusion — always compare the language of the headline with the language of the original abstract.

“The results suggest” and “research proves” are two different levels of certainty — and on the way from the lab to the newspaper headline, the first phrase very often quietly turns into the second.

Frequently asked questions

Not always — a well-designed, large randomized trial can be very strong evidence, while a systematic review made up of a few weak studies can give an unreliable result. The key is to check the quality of the methodology, not just the “review” or “study” label itself.

A narrative review describes a topic freely, without a predetermined protocol for searching and selecting studies. A systematic review uses an explicit, repeatable methodology — with a registered protocol, inclusion and exclusion criteria and a risk-of-bias assessment — which is why it’s less prone to biased selection of sources.

Not quite. A systematic review is a method of gathering and assessing studies; a meta-analysis additionally combines their numerical results statistically into a single measure. Most meta-analyses are part of a systematic review, but not every systematic review includes a meta-analysis — sometimes the studies are too different to be combined mathematically.

Fastest in PubMed — under the title of every record there’s a “Publication type” label (e.g., Randomized Controlled Trial, Systematic Review, Meta-Analysis, Case Reports). It’s a single line that settles the matter in seconds.

Yes, it’s a valuable stage of science — many therapies start with animal models. The problem arises when the media leave out this detail and write “studies show X cures Y in humans,” while the original study involved only lab mice.

Want to get better at telling solid science from one-day news?

More from our blog

Sources and further reading: J. Bohannon, “I Fooled Millions Into Thinking Chocolate Helps Weight Loss” (io9/Gizmodo, 2015); Wikipedia, “Fraudulent Lancet MMR vaccine–autism study”; L. Taylor et al., “Vaccines are not associated with autism: An evidence-based meta-analysis of case-control and cohort studies,” Vaccine, 2014; Cochrane Poland, “How to interpret systematic reviews?”; PRISMA guidelines (prisma-statement.org); the PROSPERO register (University of York); the Retraction Watch database.

Scroll to Top