Is a Positive Test Result Bad News?

Chances are hard to picture; people are easy. So think of a whole crowd.

In a crowd of 1,000 people, about 1 has the condition, and the test almost surely catches them. The other 999 are healthy, and 1 in 100 of them gets a false alarm: about 10 people. So about 11 people are ringed, and only 1 of them has it. A positive result puts you in that ringed group, and in that group the chance of having the condition is about 1 in 11.

A crowd of 1,000 is too small for exact counts (a test can’t catch 0.99 of a person), so take 100,000 people, where every count comes out whole. This is a tree diagram, as in Grade 10, but with the number of people on each branch instead of the probability: these numbers are called natural frequencies, or expected frequencies.

Read the bottom row of Figure 14.1. Everyone who tests positive is in one of the two tinted boxes: 99 people who have the condition and 999 who don’t. So

$$P(\text{has it, given a positive}) = \frac{99}{99 + 999} = \frac{99}{1{,}098} \approx 0.090,$$

about 9%. Pasang is far more likely to be one of the 999 than one of the 99.

Nothing is wrong with the test: it gives the right answer to 99,000 of the 100,000 people. But almost all of the 1,000 it is wrong about are healthy people with a false alarm, because healthy people are 999 in every 1,000 of the crowd.

Here is the whole trick in one line. “The test is 99% accurate” answers one question, and Pasang is asking a different one:

• Of the people who have the condition, how many test positive? 99%.
• Of the people who test positive, how many have the condition? 9%.

The two questions use the same words in a different order, and their answers are eleven times apart. In the notation of probability, a vertical bar is read “given”: $P(A \mid B)$ is the chance of A given that B has happened, a conditional probability. Then the two questions are

$$P(\text{positive} \mid \text{has it}) = 0.99, \qquad P(\text{has it} \mid \text{positive}) \approx 0.09.$$

Mixing them up, by forgetting how rare the condition is (its base rate), is so common that it has a name: the base rate fallacy.

You already know these two questions are different when the words are everyday ones. Almost everyone who lives in Bhaktapur is Nepali: $P(\text{Nepali} \mid \text{lives in Bhaktapur})$ is nearly 1. But only a small share of Nepalis live in Bhaktapur: $P(\text{lives in Bhaktapur} \mid \text{Nepali})$ is small. Nobody mixes those up. A test result hides the same difference behind numbers that sound medical.

Reading it from a table

Results like these are often written as a two-way table: one way across for the condition, the other for the test. Here is a different test, for a condition that 1 in 100 people have; it catches 90% of them, and gives false alarms to 5% of the rest. For 10,000 people:

	Test positive	Test negative	Total
Has it	90	10	100
Doesn’t	495	9,405	9,900
Total	585	9,415	10,000

Each conditional probability is a share of one row or one column. Given that a person has it, you are in the first row, so $P(\text{positive} \mid \text{has it}) = 90 \div 100 = 90\%$. Given a positive result, you are in the first column, so $P(\text{has it} \mid \text{positive}) = 90 \div 585 \approx 15\%$. Same corner cell, 90 people, divided by a different total.

That is the general rule. Out of everyone for whom B happens, what share also has A?

$$P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}.$$

With the table’s numbers, $\frac{90/10{,}000}{585/10{,}000}$: the 10,000s cancel, leaving 90 ÷ 585 again. Counting people and the formula are the same calculation.

The table also shows that having the condition and testing positive are not independent events. Over everyone, P(has it) = 100 ÷ 10{,}000 = 1%; among the positives it is 15%. Two events are independent when knowing that one has happened doesn’t change the chance of the other, $P(A \mid B) = P(A)$, as in Grade 10. A test is worth taking only because its result is not independent of the condition.

The tree and the table did everything by counting. Write the same counting with letters and you get a formula that works for any test. Call the base rate p (the share who have the condition), the share of them the test catches s, and the false alarm rate f (the share of healthy people who test positive). Out of N people,

• those who have it and test positive: Nps;
• those who don’t and test positive: N(1 − p)f.

A positive result means you are one of these, so

$$P(\text{has it} \mid \text{positive}) = \frac{p\,s}{p\,s + (1 - p)\,f}.$$

This is Bayes’ rule (also called Bayes’ theorem), after Thomas Bayes (below). With p = 0.001, s = 0.99, f = 0.01: $0.00099 \div (0.00099 + 0.00999) \approx 0.090$, Pasang’s 9%.

In medicine the three numbers have names. The share of those with the condition whom the test catches, s, is the test’s sensitivity. The share of healthy people it correctly clears, 1 − f, is its specificity: a false alarm rate of 1% is a specificity of 99%. The base rate is the condition’s prevalence. And the answer, the chance that a positive is real, is the positive predictive value. “99% accurate” meant a sensitivity and a specificity of 99%; the positive predictive value depends on the prevalence too.

The quick way: odds

There is a way to do Bayes’ rule in your head. Instead of a chance, use odds: the number of people who have it against the number who don’t. Before the test, the odds are 1 to 999 (written 1 : 999).

Now ask how much more likely a positive result is from someone who has it than from someone who doesn’t: s ÷ f = 0.99 ÷ 0.01 = 99 times. This number, the likelihood ratio of a positive result, belongs to the test alone. A positive result multiplies the odds by it:

$$1 : 999 \quad \xrightarrow{\ \times 99\ } \quad 99 : 999.$$

Why does multiplying work? Of N people, those who test positive are Nps who have it and N(1 − p)f who don’t, so the odds among the positives are

$$N p s : N (1 - p) f \;=\; p \times \frac{s}{f} : (1 - p),$$

the odds before the test, p:(1 − p), with the first side multiplied by s ÷ f.

Odds of 99 to 999 are a chance of 99 ÷ (99 + 999) = 99 ÷ 1,098, the 9% again. The 99 people and 999 people are the tree’s two tinted boxes. The test multiplied the odds by 99, which is a lot; but they started at 1 to 999, so after one positive they are still about 1 to 10 against.

The last answer is worth stopping on. A positive result from one and the same test means 9% in a school hall and 92% in a clinic. The test hasn’t changed; the people have. Before you can say what a result means, you must know who was tested and how common the condition is among them.

Where 1 in 100,000 people have the condition, about a thousand false alarms come with every real case, and one positive means almost nothing (0.10%). At 1 in 100, real cases and false alarms are equal in number: an even chance. At 1 in 10 it is 92%. That is why a doctor asks about your symptoms before ordering a test: the answers change the base rate, and so what the result will mean.

A better test moves the whole curve to the left. With false alarms of 1 in 1,000 (a specificity of 99.9%), a positive in the school hall means about 50% instead of 9%. For a positive to be more likely right than wrong where 1 in 1,000 have the condition, the false alarm rate has to be a little below 1 in 1,000: the test must be at least as rare in its mistakes as the condition is in the crowd.

What happens if every positive is tested a second time? Go back to the crowd at the top of the page and press Test the positives again. Of the 1,098 positives in 100,000 people, a second positive comes for 99% of the 99 who have it (98 people) and for only 1% of the 999 false alarms (10 people). Now 98 of 108 have it: about 91%.

In odds, each independent positive multiplies by the likelihood ratio again:

	Odds (has it : doesn’t)	Chance
Before any test	1 : 999	0.1%
After one positive	99 : 999, about 1 : 10	9.0%
After two positives	9,801 : 999, about 10 : 1	90.8%
After three positives	970,299 : 999, about 971 : 1	99.9%

The first positive turned 1 in 1,000 into 1 in 11; the second turned that into 10 in 11. The same multiplication, ×99, did both, but it starts each time from what the last test left.

If this was surprising, you are in good company. In 2007 the psychologist Gerd Gigerenzer gave 160 gynecologists, doctors who need exactly this number to explain a screening result, this problem at the start of a training session: 1% of the women in a region have breast cancer; a mammogram (an X-ray screening test) is positive for 90% of those who have it, and for 9% of those who don’t. A woman tests positive. Is the chance that she has breast cancer about 81%, about 9 in 10, about 1 in 10, or about 1%?

The answer, which you can now work out, is about 1 in 10 (with Bayes’ rule, 9.2%). Most of the doctors chose 90% or 81%, their answers ran all the way from 1% to 90%, and only 21% chose the right one, fewer than would have by picking at random. Then the session taught them to turn such percentages into natural frequencies: of 1,000 women, 10 have cancer, and 9 of them test positive; of the 990 who don’t, about 89 test positive. Afterwards, 87% of them chose 1 in 10.

So the difficulty isn’t in people’s heads so much as in the way the numbers are given. Percentages of percentages are hard; people are easy to count. Gigerenzer and his colleagues report the same pattern from other studies: in one from 1978, only 18% of the doctors and medical staff asked could work out the chance that a positive result was real.

In a courtroom the mix-up is called the prosecutor’s fallacy: reading “how likely is this evidence if the person is innocent?” as “how likely is the person to be innocent, given this evidence?”. A trace that matches only 1 person in 1,00,000 by chance sounds damning, but in a city of ten lakh about 10 other people would match it too (Question f).

In your inbox, the mix-up is avoided on purpose. In 2002 the programmer Paul Graham described a spam filter that learns, from mail a person has already sorted, how often each word turns up in spam and in other mail, and combines the words of a new message by Bayes’ rule into the chance that it is spam. In his mail, the word “guarantee” meant about a 98% chance of spam; he treated a message as spam above 90%. Graham reported that, on his own mail, his filter missed fewer than 5 in every 1,000 spams, with no good mail wrongly caught.

• Will Two People in Your Class Share a Birthday? Another chance that intuition gets badly wrong, worked out by the multiplication principle.
• Should You Switch Doors? A host who knows where the prize is, and a conditional probability that fooled professors.
• What Is a Game Worth on Average? Expected value, for weighing what each choice costs and gains, as in deciding whom to test.
• Can a Treatment Win in Every Group but Lose Overall? Simpson’s paradox: when who is in each group changes what the totals say.
• Is This Game Fair? (Grades 6–8) Counting equally likely outcomes to see who really has the better chance.
• Grade 10 Math: Tree diagram and Independent and dependent events, the tools this page is built on.
• Grade 12 Computing: When one kind is rare Precision, recall and the confusion matrix: the same table, for a filter.