Other · Probability

The Base Rate Fallacy

A rare disease, a test that's right 99% of the time, and a positive result. Surely you almost certainly have it? Not even close, and most doctors asked get it wrong too. Four steps: make your guess, line up ten thousand people, take the test again, then turn the dials.

1

You test positive

A disease affects 1 in every 1,000 people. There's a test for it that is 99% accurate: it catches 99% of people who have it, and gives a false alarm to only 1% of people who don't. You take the test, and it comes back positive. How likely is it that you have the disease?

1 in 1,000people have the disease
99% caughtif you're sick, the test says positive
1% false alarmif you're healthy, it still says positive
Your guess
2

Line up ten thousand people

Forget percentages and count people instead. Here are 10,000 people, one square each. One in a thousand is sick, so ten of them have the disease (the glowing ones). Now give every one of them the test.

Sick Healthy
Tested positive
?
Chance a positive result means you're sick
?
3

Now take the test again

Here are just the 110 people who tested positive. Everyone gets a second, independent test. The ten sick people test positive again. The hundred healthy ones each have the same 1% chance of another false alarm, so about one of them is unlucky twice.

Sick, positive every time Healthy, still positive Healthy, cleared
After one positive
9%
10 sick out of 110 positive
After two positives
?
After three positives
?
4

Turn the dials

Make the disease rarer or more common, and the test better or worse, and see what a positive result is worth. The test is just as accurate on sick and healthy people here, as in the puzzle.

In a town of 100,000: 100 sick, and the test catches 99 of them. It also flags 999 healthy people. So of everyone who tests positive, 99 of 1,098 are really sick.

Chance you're sick after one positive9%
Chance you're sick after two positives91%
Chance you're sick after three positives99.9%

How rare it is matters as much as how good the test is.

A test result doesn't tell you your chance of being sick on its own. It updates the chance you started with. Start from 1 in 1,000 and even a 99% accurate test can only get you to about 9%. Start from 1 in 10, because you have symptoms or were exposed, and the same positive means about 92%.

This is why doctors don't screen everyone for rare diseases, and why a first positive is followed by a second, different test before anyone acts on it.

Worth knowing

What it's called

The base rate fallacy

Also known as the false positive paradox. The base rate is how common something is before you look for it, here 1 in 1,000. Ignoring it is the mistake; Bayes' theorem, which combines the base rate with the evidence, is the fix.

Doctors get it wrong too

Harvard Medical School, 1978

Casscells and colleagues put a version of this puzzle to 60 staff and students, with a 5% false alarm rate. Almost half answered 95%. The right answer is about 2%, and only 11 of the 60 gave it.

Why anyone cares

Every alarm for a rare thing

A fraud model that is 99% accurate, on payments where 1 in 1,000 is fraud, flags about ten honest payments for every real one. The same holds for security alerts and anomaly detectors. That's why they are judged on precision, not accuracy, and why a flag starts a second check rather than a block.

Every number is worked out in your browser from the rates on the page. The crowd is rounded to whole people to keep the picture readable; the percentages are exact. The test is taken to be equally accurate on sick and healthy people, as in the classic version of the puzzle.

Try another puzzle →