How to check whether a statistic in an article was taken out of context

Guide · Numbers and data · Check it yourself Dark graphic: a line chart with a highlighted fragment labelled +18%?, a note 6 → 7 in 100 people and a magnifying glass, a symbol of statistics taken out of context
⏱ approx. 11 min read 📚 Category: Blog 🕓 Updated: November 22, 2026

Short answer: go back to the original source and look for what the number leaves out: the baseline (an 18% higher risk of bowel cancer from processed meat means roughly 6 to 7 cases per 100 people, according to Full Fact), the absolute numbers behind a percentage, the time window, the definition and whether an average hides very different groups, as in Berkeley’s 1973 admissions data.

Numbers feel objective. That’s exactly why they’re so effective when they’re taken out of context: the figure itself may be perfectly accurate while the impression it creates is wrong. A statistic is like a photo crop. To judge it, you need to see what was left outside the frame.

Trap 1: relative risk without absolute risk

In October 2015 the World Health Organization’s cancer agency (IARC) announced that “each 50 gram portion of processed meat eaten daily increases the risk of colorectal cancer by 18%”. Headlines ran with the 18%. Many readers understood it as an 18% chance of getting cancer, or as an effect comparable to smoking.

The 18% was a relative increase: a rise compared with the risk people already have. UK fact-checker Full Fact put it in absolute terms: if 100 people ate a roughly 50 g bacon sandwich every day of their lives, you’d expect bowel cancer cases in that group to rise from about 6 to 7. Full Fact also rated as inaccurate the claim that WHO had said processed meat is as big a cancer threat as cigarettes: both are classified as proven causes of cancer, but the size of the risk is very different.

Rule: whenever you see “increases the risk by X%”, ask: X% of what? What’s the baseline risk, and how many extra cases per 100 or per 10,000 people does it mean?

Trap 2: the average that hides the groups (Simpson’s paradox)

In 1973 the University of California, Berkeley, was worried about bias in graduate admissions. Of 12,763 applicants, 8,442 were men and 4,321 women, and overall roughly 44% of men were admitted compared with about 35% of women. That looks like clear discrimination.

When statisticians Peter Bickel, Eugene Hammel and J. W. O’Connell looked department by department (published in Science in 1975), the picture changed: in most departments, women’s admission rates were equal to or higher than men’s. The explanation was that women applied more often to highly competitive departments with low admission rates for everyone. The overall figure was accurate, but combining very different groups produced a misleading conclusion. This is Simpson’s paradox: a trend that appears in pooled data can disappear or reverse when the groups are examined separately.

Ask: is this a single average for very different groups (regions, age groups, departments, product types)? What do the subgroups show?

Seven more ways a number loses its context

Common statistical traps and how to spot them (our overview)
TrapExample (illustrative)What to check
Cherry-picked time window“Crime up 40% since last spring” (starting from an unusually low month)A longer time series; is the start point typical?
Small base“Cases up 200%!” (from 2 to 6)The absolute numbers behind the percentage
Percent vs percentage pointsUnemployment from 5% to 6% reported as “up 1%” or “up 20%”Both are correct; which one is the article using and why?
Average vs median“Average salary is €5,000” (pulled up by a few very high earners)The median and the distribution
Truncated axisA chart starting at 95 makes a 2-point change look hugeWhere the y-axis starts
Changed definitionA “surge” after the way something is counted changedMethodology notes, breaks in the series
Correlation as cause“Towns with more cafés have higher incomes, so cafés make people rich”Other explanations, study design

Sample size deserves its own mention: a striking percentage from a survey of 40 people says much less than the same percentage from 4,000. We explain why in sample size and confidence intervals.

How to check a statistic in 8 steps

1

Find the original source

Follow the link or search for the report, study or dataset the number comes from. A number without a source is a claim, not data.

2

Read the definition

What exactly was measured, and how? Unemployment, crime or poverty can be defined in several ways.

3

Get the absolute numbers

Behind every percentage there’s a “from what to what”. Small bases produce dramatic percentages.

4

Check the baseline

For risks: what’s the risk without the factor? Convert relative risk into extra cases per 100 or 10,000 people.

5

Widen the time window

Look at a longer series. Was the starting point unusual? Is the trend real or seasonal?

6

Split the average

Ask whether different groups are pooled. Look at subgroups, medians and distributions.

7

Separate correlation from cause

Ask what else could explain the link and whether the study design allows causal conclusions.

8

Compare with another source

Check the same figure in an independent dataset or with a statistician’s or fact-checker’s analysis.

Worked example (illustrative)

Imagine a headline: “Electric scooter accidents up 150% in one year!”

  1. Source: the article cites the city’s road safety report. You open it.
  2. Absolute numbers: 12 accidents last year, 30 this year. A real increase, but a small base.
  3. Context: the number of scooters for rent rose from 500 to 2,000 in the same period. Per 1,000 scooters, accidents actually fell from 24 to 15.
  4. Definition: from this year, the police also record incidents without injuries. Part of the rise is a change in counting.

None of the facts in the headline was false, yet it suggested that scooters became more dangerous, while the full data suggest the opposite. That’s what “out of context” means.

Rates beat counts: when the size of the group changes, compare rates (per 1,000 users, per 100,000 residents, per million kilometres), not raw counts.

Phrases that should make you look for context

  • “up X%” or “doubles the risk” without a baseline;
  • “record”, “highest ever”, “since [unusual year]”;
  • “on average” for a very diverse group;
  • “linked to”, “associated with”, presented as cause;
  • a chart without axis labels or a source;
  • a number with no date, no source and no definition.

More on checking where numbers come from: “studies show”: one study or a systematic review? and searching official data.

Frequently asked questions (FAQ)

The number may be accurate, but the information needed to interpret it, such as the baseline, time period, definition or group breakdown, is missing, so it creates a misleading impression.

Relative risk compares two groups (e.g. 18% higher), while absolute risk tells you the actual chance or number of cases (e.g. 7 instead of 6 per 100 people). You need both to understand the size of an effect.

No. Full Fact rated that claim inaccurate: both are classified as causes of cancer, but the size of the risk from processed meat is far smaller.

A situation in which a trend in combined data disappears or reverses when the data are split into groups, as in Berkeley’s 1973 graduate admissions.

If a rate rises from 5% to 6%, it went up by 1 percentage point, which is a 20% relative increase. Both can be correct, but they sound very different.

An average can be pulled up or down by a few extreme values. The median shows the middle value and is often a better picture of a typical case.

Check where the y-axis starts, whether the scales are labelled, whether the time range is cut and whether the source is given.

They are much less certain. The same percentage from 40 people and from 4,000 people carries very different uncertainty.

National statistics offices, Eurostat, the OECD, WHO and the original study or report cited in the article.

It can explain concepts or help with calculations, but it can also invent numbers. Always verify the figures in the original source.

More from our blog

Sources: Full Fact: Processed meat isn’t as big a cause of cancer as smoking (2015) · UICC: How to interpret IARC findings on red and processed meat · EUFIC: Absolute vs. relative risk · Bickel, Hammel, O’Connell: Sex Bias in Graduate Admissions: Data from Berkeley (Science, 1975) · HKU Critical Thinking: Simpson’s paradox · The refsmmat report: Simpson’s paradox and statistical urban legends. The trap overview, worked example, steps and phrase list are our own analysis. Figures for the Berkeley case vary slightly between sources depending on how departments are counted.

Scroll to Top