Skip to content

← Thinking Clearly

Rare things and good tests

Chapter 16 · Numbers without heavy math

In this chapter 5 sections
  1. Counting women instead of chances
  2. A real test, counted
  3. The 99% test
  4. Who was tested
  5. You’ve won

At the start of a training session in 2007, Gerd Gigerenzer handed 160 gynecologists a short problem. They were doctors who look after women’s health, and the problem was one they meet at work. In a certain region, it said, 1% of women have breast cancer. An X-ray screening test for it, a mammogram, comes out positive for 90% of the women who have the cancer, and for 9% of the women who don’t. A woman tests positive. She wants to know whether that means she has cancer for certain, or what the chances are.

He gave them four answers to choose from: about 81%, nine in ten, one in ten, and about 1%.

Most of the doctors chose nine in ten or 81%. Their answers ran all the way from 1% to 90%. Only 21% of them picked one in ten, which is the right answer, and fewer than would have got it by choosing at random from the four. Then Gigerenzer showed them a different way of putting the same facts, and asked again. This time 87% of them got it.

The different way is the whole of this chapter, and it takes about a minute to learn. It also explains why a test that sounds nearly perfect can be wrong most of the times it says yes, and why a message saying you’ve won a prize is nearly always a lie.

Counting women instead of chances

Here is the same problem, told as people instead of percentages. Think of 1,000 women in that region, all screened.

Ten of them have breast cancer. That is the 1%. Of those ten, the test finds nine. That is the 90%.

The other 990 women don’t have cancer. The test wrongly flags 9% of them, which is about 89 women.

So 98 women in all get a positive result: 9 who have cancer, and 89 who don’t. A woman holding a positive result is one of those 98, and only 9 of them have cancer. That’s about one in eleven, or close to one in ten.

Nothing in the arithmetic is harder than counting. What made the doctors stumble was the way the numbers were given: as chances of chances, each one a percentage of a different group. Turned into counts of people out of one crowd (Gigerenzer calls them natural frequencies), the answer is almost there on the page.

The reason it comes out so low is the size of the two groups. The women with cancer are few, and the test catches nearly all of them. The women without cancer are many, and the test catches only a small share of them by mistake. But a small share of a big crowd can be more people than nearly all of a small one. Here the false alarms outnumber the real cases about ten to one.

It helps to see that two different questions hide in the problem, worded almost alike. Of the women who have cancer, how many test positive? Nine in ten. Of the women who test positive, how many have cancer? About one in ten. The doctors who answered nine in ten had given the right answer to the first question when they had been asked the second. People make the same switch with words they know well, though rarely with things they can picture. Nearly every elephant is gray. Very few gray things are elephants.

A real test, counted

Those numbers were a teaching problem, rounded for easy counting and not far from what studies of the time had measured. Real screening has been counted too. Between 2007 and 2013, the Breast Cancer Surveillance Consortium, a group of research registries in the United States, followed nearly 1.7 million screening mammograms, and checked which women were found to have breast cancer in the year after. Shared out among 1,000 screenings, this is what happened.

5 have breast cancer, and it was found1 has breast cancer, and it was missed111 are called back, but have no cancer883 are told all is clear, and have noneCalled back: 116, and 5 of them have cancer
Figure 16.1 One thousand screening mammograms in the United States, 2007 to 2013, each square one woman screened. The dashed line goes round those called back for more tests.

About six women in every thousand had breast cancer. The X-ray picked out five of them and missed one. Of the 994 without cancer, it cleared 883 and called back 111. So 116 women were called back, about one in nine of everyone screened, and five of those had cancer. A call-back meant cancer for about one woman in 23.

And yet by the usual measures this is a good test. It found 87% of the cancers and cleared 89% of the women who had none. Both numbers are true, and both answer the first question. The woman with the letter in her hand is asking the second.

None of this means the X-ray failed. A call-back isn’t a diagnosis. It leads to a closer look, more pictures and sometimes a small sample of tissue, and for most women the closer look clears them. The screening test is a sieve, and its job is to let very few cancers through, which it mostly does. The cost of that is a lot of worry for women who turn out to be fine, and it adds up over the years. One American study worked out that of women screened every year from the age of 40, more than half would be called back at least once in ten years for something that wasn’t cancer. Whether screening is worth that, and from what age, is something doctors and health services weigh carefully, and countries decide it differently. The numbers say only what a call-back means, and that’s worth knowing before the letter comes.

The 99% test

The rarer the thing, the starker it gets. Imagine a test that is right 99% of the time, both ways: it says yes to 99 in every 100 people who have some condition, and no to 99 in every 100 who don’t. Imagine too that one person in a thousand has the condition. (Both numbers are made up, to keep the counting easy.)

Test 1,000 people. The one person who has it almost surely gets a yes. Of the 999 who don’t, 1% get a yes anyway, which is about ten people. So about 11 people are told yes, and one of them has the condition. A test that is 99% accurate is wrong, here, ten times out of eleven when it says yes.

The test hasn’t got worse. It’s still right about 990 of the 1,000 people. Almost all its mistakes fall on healthy people, simply because almost everyone is healthy. And a no from it is very nearly always right, for the same reason: hardly anyone it says no to had the condition to begin with.

For something rarer still, even a superb test needs help. In a study published in 1998, a man who belonged to none of the groups at special risk of HIV, the virus that causes AIDS, went to 20 public health centers in Germany and asked for a test at each one. He was working undercover for Gigerenzer’s team. In the talk before each test he asked a plain question: could he test positive without having the virus? Thirteen of the 20 counselors, mostly doctors, told him no, in words like “absolutely impossible”, “with absolute certainty, no” and “the test is absolutely certain”. One said, “No, only in France, not here.” Only three said straight away that it could happen, though rarely.

It could. The test of those years was excellent, wrong about a healthy person perhaps once in 10,000 times. But among men like him, perhaps one in 10,000 had the virus. So out of 10,000 such men, about one would be infected and test positive, and about one more would test positive without being infected. For a man at low risk, a positive result could be about as likely to be a mistake as not.

That’s why the World Health Organization’s advice never rests an HIV diagnosis on a single test. Since 1997 it has advised countries where fewer than one person in 20 has HIV to give a diagnosis only after three positive tests in a row, and countries where the virus is more common, two. Since 2019 it has urged countries to move toward three. The rarer the thing, the more a yes has to be checked.

Who was tested

A test’s yes, then, doesn’t carry a fixed meaning. It depends on who was tested, because that decides how many of the people in the crowd had the thing to begin with.

Take Gigerenzer’s X-ray again, with the same 90% and 9%, but use it on 1,000 women who came to a doctor because they had found a lump, and suppose that one in ten of them has cancer. (That figure is made up for the example.) Now 100 have cancer and the test finds 90. Of the 900 who don’t, 9% is 81. So 171 test positive, and 90 of them have cancer: about half. In the screening crowd the same yes meant about one in ten.

This is why a good doctor asks questions before ordering a test, and why a test that is useful in a clinic can mislead when it is given to everyone. The questions change the crowd. A test given to people who already have the signs of an illness says yes mostly to the ill. The same test given to a whole town says yes mostly to the well.

So when anyone tells you a test is 99% accurate, or 90%, or 87%, two more numbers are needed before the yes means anything. How common is the thing in the people being tested? And how often does the test say yes to people who don’t have it? With those, you can count your own crowd of a thousand. (The working, with a crowd to test on screen, is in Math, Going Further’s Is a Positive Test Result Bad News?)

You’ve won

The same counting works on a phone.

A message arrives saying you’ve won Rs 25 lakh in an online lottery. To claim it, you need only pay a small fee first, for tax or customs. Nepal Police warned about exactly these messages in 2018, sent by phone, Viber and WhatsApp from inside Nepal and abroad, along with promises of parcels of gadgets and jewelry. They had recorded over 500 complaints against lottery and parcel scammers in the past year. In 2023 The Kathmandu Post told the story of a man who got a message on Facebook Messenger saying he had won Rs 2.5 million and should contact an agent in India. To release the prize he paid Rs 75,000 in three installments, from three different banks, and then he was asked for a final Rs 100,000.

Think of the message as a test. It says yes, you’ve won. How often does it say yes to someone who really won?

Scammers don’t send it to winners. They send it to as many phones as they can, because sending costs them almost nothing and they need only a few to pay. So the message says yes to nearly everyone it reaches. That’s a test with a false alarm rate close to 100%. Now count the crowd: everyone in Nepal who got a “you’ve won” message this year. The real winners in that crowd, people who entered a real draw and are being told by the people who ran it, are very few, while the scammers’ messages go to every phone they can reach. Like the X-ray’s 111 false alarms around five real cases, the fakes bury the real ones, only far more so.

If you never entered the lottery, the counting is simpler still. Nobody wins a draw they weren’t in, so every such message is false. And a real prize doesn’t ask for money before you get it. Nepal Police’s advice is not to answer such messages at all, and to report them. (What to do, step by step, is in Safe Online, in Practice’s Ignore prizes you never entered.)

The trick works on people for the same reason the X-ray problem fooled doctors. The message talks only about you, one person, and a single person is easy to imagine as the lucky one. It never mentions the crowd. Ask how many people got the same message, and how many of them won, and the prize disappears.

So there are two plain questions for any yes, from a test, a message, an alarm or a sign. Out of how many? And of those who got this yes, how many were real? Behind both is the number the yes never mentions: how common the thing was before anyone looked. A yes from a good test moves you toward it. How far it should move you depends on where you started, and that question runs much further than tests.

Found a mistake? Please tell us.