Caveat Scientia
Knowledge comes with a warning label
Science Literacy · Foundations
Why That Scary Statistic Might Not Mean What You Think
Headlines weaponise numbers. Learn to disarm them — and see what the math actually says about risk, probability, and the statistics that dominate your newsfeed.
Every morning, the headlines deliver fresh catastrophe. Bacon causes cancer. One glass of wine raises your risk of dying. A new study has overturned everything we knew about sunscreen. By lunchtime you’ve been told to fear your breakfast, your commute, and your tap water — all before anyone has mentioned what “risk” actually means in any of these claims.
The numbers aren’t necessarily wrong. But numbers stripped of context are a different thing entirely from numbers that inform. This piece is about the gap between those two things — and the statistical sleight-of-hand that turns modest findings into viral alarm.
“This food increases cancer risk by 50%!”
“New drug cuts your risk of heart attack by 40%!”
“One drink a day raises your chance of dying by 20%!”
These headlines are dramatic, urgent, and almost universally misunderstood. Let’s take a closer look at the math behind the hype.
Relative Risk vs. Absolute Risk: The Most Important Distinction You’ve Never Been Taught
There are two ways to report a change in risk. One sounds dramatic. The other is honest. Most headlines use the dramatic one.
Absolute risk is the actual probability that something happens to a specific group of people — for example, 5 out of every 100 people develop a particular condition. Relative risk compares that probability between two groups — for example, people who eat X are 20% more likely to develop it than people who don’t.
Here’s the problem: relative risk, in isolation, is almost meaningless without knowing the baseline absolute risk. Consider this scenario:
Imagine a headline reads: “Eating processed meat increases colon cancer risk by 20%.” That sounds alarming. But let’s do the arithmetic.
A 20% relative increase on a 5% baseline produces a new risk of 6% — an absolute increase of just one percentage point. The headline is technically accurate. But it conveys something very different from the reality it describes.
The same principle applies in the positive direction. A drug that claims to reduce heart attacks by 50% sounds miraculous. But if it reduces risk from 2% to 1%, the absolute risk reduction is just 1 percentage point — meaning 100 people must be treated for one to benefit.
Relative risk sounds dramatic. Absolute risk reduction tells you what actually changes for real people. Always ask: “50% of what, exactly?”
The Base Rate Fallacy: Ignoring the Starting Point
Even when a test or statistic is highly accurate, ignoring how common (or rare) a condition actually is can lead to wildly wrong conclusions. This is the base rate fallacy — and it’s one of the most counterintuitive traps in probability.
Imagine a blood test that detects a rare disease with 99% accuracy. If you test positive, most people would assume they almost certainly have the disease. But watch what happens when you account for the base rate:
Suppose only 1 in 1,000 people have the disease. Test 10,000 people:
Of 108 positive tests, only 9 are genuine cases. Your actual probability of having the disease, given a positive result, is roughly 8.3% — not 99%.
“A 99% accurate test, applied to a rare condition, is wrong more often than it’s right.”
Caveat ScientiaThis is why rare disease screening programmes require careful design, follow-up testing, and honest communication about what a positive result really means. The accuracy of the test is only part of the story. The prevalence of the condition in the tested population is the other part — and it’s usually left out of the headline.
Rare conditions remain rare even with high-risk multipliers. Base rate matters as much as test accuracy. A positive result for a rare condition is still more likely to be wrong than right.
Correlation Is Not Causation
Two things can rise and fall together without one causing the other. This seems obvious when stated plainly — yet headlines routinely imply causation from data that only demonstrates association.
A study might show that people who drink more diet soda have higher rates of obesity. Does diet soda cause obesity? Not necessarily. It’s far more likely that people already trying to manage their weight are more likely to choose diet drinks — a phenomenon called reverse causation. Or that some third variable — overall diet quality, sedentary lifestyle — is driving both patterns. These are confounding variables, and accounting for them properly is one of the hardest problems in epidemiology.
Confounders don’t make a study wrong. They make it incomplete — which is why replication across different populations and study designs matters so much. A single study that finds a correlation is a prompt for further investigation, not a mandate to change behaviour.
A link between two things is not evidence that one caused the other. Look for randomised controlled trials and replicated findings before assuming cause and effect.
What “Statistically Significant” Actually Means
If you’ve read any science reporting in the last decade, you’ve seen the phrase “statistically significant.” It’s used as a stamp of approval — proof that the result is real, important, and worth acting on. It is none of these things, at least not reliably.
Statistical significance is measured using a p-value. The p-value tells you the probability that you’d see a result at least this extreme if there were actually no effect at all. By convention, a p-value below 0.05 is deemed “significant” — meaning there’s less than a 5% chance the result is pure chance.
Here’s what that doesn’t mean:
- It doesn’t mean the effect is large or clinically meaningful.
- It doesn’t mean the finding will replicate.
- It doesn’t mean there’s only a 5% chance the hypothesis is false.
- It doesn’t account for how many hypotheses the researchers tested before finding this one.
That last point deserves attention. If researchers test 20 different variables, simple probability predicts that one of them will appear “significant” by chance alone — even if none of them actually matter. This is sometimes called p-hacking or the multiple comparisons problem, and it’s a known driver of unreplicable findings in the published literature.
The scientific community is actively grappling with this. Some researchers advocate raising the significance threshold to p < 0.005; others argue for abandoning the binary threshold altogether in favour of reporting effect sizes and confidence intervals. For now, treat any single “statistically significant” finding as a starting point — not a conclusion.
“Statistically significant” does not mean practically significant. A large study can make a tiny, meaningless effect look significant. Always look for the effect size — not just the p-value.
How to Read a Risk Statistic: A Field Guide
The next time a headline delivers a number that makes you want to throw out your fridge contents, run through these questions before you react.
- Is this relative or absolute risk?
A 50% increase in a 1% risk is still just 1.5%. Ask for the baseline and the actual numerical change. - What’s the base rate?
Rare conditions remain rare, even with frightening-sounding multipliers. The frequency of a condition in the population is as important as any test’s accuracy. - Is causation being claimed or just correlation?
Look for the study type. Observational studies identify associations. Only well-designed randomised controlled trials can reliably establish causation. - What does “statistically significant” actually mean here?
Check whether an effect size is reported. A significant p-value on a trivially small effect is not news. One study rarely is. - How large was the study, and who was in it?
Larger, more diverse studies produce more reliable results. A finding from 200 people in one country may not generalise to anyone else. - Has the finding been replicated?
Science moves by consensus across multiple independent studies, not by individual breakthroughs. Be especially cautious of the phrase “first study to show.”
Caution Over Clickbait
Science is a slow, self-correcting process. It thrives on careful interpretation, replication, and the willingness to be wrong. Headlines do not have these qualities. They have a different job: to make you stop scrolling.
The mismatch between those two things — rigorous science and attention-hungry communication — is where most public misunderstanding of risk is born. The numbers themselves are often accurate. What’s missing is context: the baseline, the effect size, the study design, the number of comparisons, the replication record.
None of this means you should dismiss scientific findings. It means you should hold them at arm’s length until the picture is fuller. Ask what the statistic is compared to. Ask who benefits from framing it this way. Ask how many times the finding has been reproduced.
The warning label on knowledge isn’t a reason to stop reading. It’s a reason to read better.
