The Base Rate Fallacy
A rare disease, a test that's right 99% of the time, and a positive result. Surely you almost certainly have it? Not even close, and most doctors asked get it wrong too. Four steps: make your guess, line up ten thousand people, take the test again, then turn the dials.
You test positive
A disease affects 1 in every 1,000 people. There's a test for it that is 99% accurate: it catches 99% of people who have it, and gives a false alarm to only 1% of people who don't. You take the test, and it comes back positive. How likely is it that you have the disease?
Line up ten thousand people
Forget percentages and count people instead. Here are 10,000 people, one square each. One in a thousand is sick, so ten of them have the disease (the glowing ones). Now give every one of them the test.
Now take the test again
Here are just the 110 people who tested positive. Everyone gets a second, independent test. The ten sick people test positive again. The hundred healthy ones each have the same 1% chance of another false alarm, so about one of them is unlucky twice.
Turn the dials
Make the disease rarer or more common, and the test better or worse, and see what a positive result is worth. The test is just as accurate on sick and healthy people here, as in the puzzle.
In a town of 100,000: 100 sick, and the test catches 99 of them. It also flags 999 healthy people. So of everyone who tests positive, 99 of 1,098 are really sick.
How rare it is matters as much as how good the test is.
A test result doesn't tell you your chance of being sick on its own. It updates the chance you started with. Start from 1 in 1,000 and even a 99% accurate test can only get you to about 9%. Start from 1 in 10, because you have symptoms or were exposed, and the same positive means about 92%.
This is why doctors don't screen everyone for rare diseases, and why a first positive is followed by a second, different test before anyone acts on it.
Worth knowing
The base rate fallacy
Also known as the false positive paradox. The base rate is how common something is before you look for it, here 1 in 1,000. Ignoring it is the mistake; Bayes' theorem, which combines the base rate with the evidence, is the fix.
Harvard Medical School, 1978
Casscells and colleagues put a version of this puzzle to 60 staff and students, with a 5% false alarm rate. Almost half answered 95%. The right answer is about 2%, and only 11 of the 60 gave it.
Every alarm for a rare thing
A fraud model that is 99% accurate, on payments where 1 in 1,000 is fraud, flags about ten honest payments for every real one. The same holds for security alerts and anomaly detectors. That's why they are judged on precision, not accuracy, and why a flag starts a second check rather than a block.
Every number is worked out in your browser from the rates on the page. The crowd is rounded to whole people to keep the picture readable; the percentages are exact. The test is taken to be equally accurate on sick and healthy people, as in the classic version of the puzzle.
Try another puzzle →