Simpson's Paradox
One treatment cures more patients with small kidney stones, and more patients with large ones. The other cures more patients overall. Both are true, from the same data.Three steps: pick a treatment, see who was given each one, then see why the right answer depends on a question the numbers can't answer.
Pick a treatment
You have kidney stones. A hospital has tried two treatments on 350 patients each and counted how many were cured. These are real numbers from a 1986 study. Which one do you want?
| Patients | Treatment Aopen surgery | Treatment Bkeyhole surgery |
|---|---|---|
| All patients | 78%273 of 350 cured | 83%289 of 350 cured |
Pick one. There is a second way to read these numbers once you do.
Who got which treatment
Here is every patient as a dot. Lit means cured. Look at who each treatment was given to: large stones are much harder to cure, and doctors kept the big operation, A, for the hard cases.
Give both the same patients
Each line is how often a treatment would cure people for a given mix of small and large stones. The two dots are the mixes they actually got. Slide to hand both treatments the same mix.
So which numbers do you trust?
Here, the split. But it would be a mistake to learn “always split the data”. The very same table can call for either answer, depending on a question the numbers can't answer: what caused what.
When the split came first
Stone size was settled before anyone chose a treatment, and it pushed both ways: it steered doctors towards A, and it made a cure less likely. A thing that drives both the choice and the outcome muddies the totals, so you compare like with like, small with small and large with large.
When the treatment caused the split
Suppose instead a pill prevents heart attacks by lowering blood pressure, and you split patients by their blood pressure after taking it. Now the groups are made by the pill itself. Splitting hides the very thing it does, and the totals are the honest comparison.
Numbers say what goes with what. Not why.
Nothing in the table itself tells you which reading is right. Relabel the kidney-stone table with the blood-pressure story and the numbers stay identical, but the right answer flips to the totals. The answer comes from knowing how the data came about: who chose, what came first, what caused what.
Before trusting an average, ask what it is averaging over, and whether that mix is the same on both sides. A model can get better in every customer segment and still look worse overall, simply because this month's traffic leaned towards the hard segments.
Worth knowing
Older than its name
Edward Simpson described it in 1951, though Karl Pearson and Udny Yule had run into it around 1900. The kidney-stone numbers are from a 1986 study in the British Medical Journal comparing open surgery with keyhole techniques.
Berkeley admissions, 1973
Graduate admissions at UC Berkeley took 44% of men who applied and 35% of women, which looked like bias. Department by department, most admitted women at a similar or higher rate than men. Women had mostly applied to the departments that turned most people away.
Every average hides a mix
Conversion rates, model accuracy, claim costs, hospital league tables: each is an average over a mix that can shift underneath it. When a number moves, check whether the thing got better or worse, or just the mix of what it was measured on.
Patient counts are from Charig and colleagues, BMJ, 1986. The chart in step 2 keeps each treatment's cure rate for small and large stones fixed and only changes the mix of patients.
Try another puzzle →