Skip to content

← Thinking Clearly

Stories and what's missing

Chapter 10 · Where a claim comes from

In this chapter 4 sections
  1. Holes in the planes
  2. The crowd behind the post
  3. Toppers and the cousin abroad
  4. Who isn’t in the picture

The Roman writer Cicero, in a book about the gods written more than two thousand years ago, tells a story about a man called Diagoras, known as the Atheist. Diagoras was on the island of Samothrace, and a friend there tried to change his mind. You think the gods don’t care what happens to people, the friend said. Then what about all these paintings? Each one had been offered by someone caught in a storm at sea who had made a vow to the gods and come safely to port.

Diagoras had a short answer. There are no paintings, he said, of the people who were shipwrecked and drowned.

An old oil painting of a three-masted sailing ship tossed on dark, stormy waves, with a picture of the Virgin Mary in the clouds at the top corner and a line of writing along the bottom.
Figure 10.1 A painting given in thanks at Genoa in 1881, for a ship’s officer brought through a hurricane off North America that June. A ship that sank sent no painting. Photo: Marcello Baschenis, Public domain

Every painting on the friend’s wall was real, and every sailor in them really had come home. The trouble was the wall itself. It could only ever hold one kind of story, because only one kind of sailor was left to bring one. Anyone who made a vow and drowned anyway left nothing behind. So the wall would have looked just as full whether the gods saved sailors or not.

That’s the whole of this chapter in one line: when the stories reach you only if things went well, counting the stories tells you almost nothing about how often things go well. A hundred happy stories can come from a hundred tries or from a hundred thousand. People call this survivorship bias, since the ones left to be counted are the survivors. Nobody has to lie for it to happen, and nobody has to pick the best cases on purpose. The world does the picking.

Holes in the planes

The case every book on this subject tells comes from the Second World War. In the summer of 1942 a group of mathematicians and economists came together at Columbia University in New York to work on problems for the American military. They called themselves the Statistical Research Group. Two of them later won Nobel Prizes in economics. One of them was Abraham Wald, a mathematician born in 1902 in Kolozsvár, the city now called Cluj, in Romania. As a Jew he had lost his place in Vienna, and in 1938 he had come to America.

Bombers came back from their missions full of holes, and in 1943 Wald wrote a set of eight memos on the question underneath them: what can the damage on the planes that came back tell you about the planes that didn’t? To show how his method worked, he made up an example. Four hundred planes fly a mission, and 359 come back. Between them the planes that returned carry 202 hits. The engines make up a little over a quarter of what an enemy gunner can see of a plane. If hits landed evenly, then, about a quarter of the 202, around 54, should be in the engines. Only 39 are.

Where did the missing engine hits go? Down, with the 41 planes that never came back. A hit in an engine was more likely to bring a plane down, so engine hits were scarce among the planes left to be counted. Working through his made-up numbers, Wald found that one hit in an engine brought a plane down about 39 times in 100, and one hit in the body of the plane about 5 times in 100. The body was full of holes on the returning planes precisely because a hit there was usually something a plane could survive. The holes you could see showed where a plane could be hit and still get home.

Wald also put his finger on the hardest part. Suppose, he wrote, that every hit on every returning plane were in one area. That could be explained in two ways: the other areas were never hit, or a hit there always brought the plane down. From the damage to returning planes alone, he wrote, it was “impossible” to tell which. He needed one more thing from outside: where hits land on planes in general. That, he wrote, could be known from how the enemy’s guns scatter their fire, or found by firing dummy bullets at a plane and recording where they hit.

That second half matters as much as the first. Noticing that some cases are missing isn’t enough. To say what the survivors mean, you have to know something about the whole crowd that set out: how many there were, and what would have happened to them if nobody had sorted them.

The crowd behind the post

Take the forwarded video, the made-up one in which a grandmother’s diabetes is cured by हिमाली बुटी चिया. Suppose her story is perfectly true. She drank the tea, and she got better.

Now picture everyone who bought the tea. Some of them got better too, for reasons of their own: an illness that eased, a change in what they ate, a doctor’s medicine that began to work at last. Those people have a story to tell, and a reason to tell it. Someone who drank the tea and saw no change has much less to say, and may not want to say they paid for it. Then the seller chooses the best of what was said for the video. Each step throws away some of the people who didn’t get better, and none of the steps needs anyone to lie.

Try it

Who Speaks Up

A thousand made-up people drink a tea that does nothing. One who gets better posts about it 20 times as often as one who doesn’t. Change the view, then move the slider.

On screen: 1,000 of 1,000got betterdid not
Got better, of everyone
31%
Got better, of those on screen
31%

Everyone who bought the tea. The green ones got better, though the tea did nothing.

When 30 in 100 would get better anyway and when only 10 in 100 would, the crowds are completely different: in the second, less than a third as many people get better. The posts change much less: even at 10 in 100, about four in five of them are happy. The video doesn’t change at all. It still shows ten happy faces, because ten were all the seller ever needed, and there were always ten to find.

That’s why a pile of stories, however true each one is, can’t tell you whether the tea works. None of this says it doesn’t. It says the stories can’t settle it, because they would look much the same either way. To know, you need everyone who drank it, the ones who said nothing included, and you need to know what would have happened to them without it. The way to find that out is a fair test, which this book comes to later.

Toppers and the cousin abroad

In 2014 The Kathmandu Post described how the centers that prepare students for the medical entrance exam celebrated each year with newspaper advertisements and billboards covered in photos of students who had won places in medical colleges. The same report said that around 9,000 students joined the three preparation centers, and that there were around 2,500 MBBS places in all. So even if every one of those places had gone to their students, most of them could never have been on a billboard. Nobody puts up a photo of a student who didn’t get in.

The faces on a billboard prove that a center had some successful students, which nobody doubted. What a student choosing a center wants to know is different: out of everyone who joined, how many got in, and how many of them would have got in anyway? The strongest students may simply go to the best-known center. The billboard can’t answer either question.

Sometimes the faces don’t even mean what they seem to. In India, a coaching company advertised “10 in Top 10 selections” for the 2020 civil service exam, with the ten successful candidates’ names and photos. India’s consumer protection authority looked into it. The ad named the course the top candidate had taken, a full course costing 1,40,000 Indian rupees, and said nothing about the other nine. Only one of them had taken a full course. Six had bought a set of practice tests, and two had taken a mock test that cost 750 rupees. In January 2025 the authority fined the company 3 lakh rupees, and noted that several coaching companies put the same successful candidates’ names and photos in their own advertisements.

The same filter works inside families. The cousin who comes home from the Gulf with gifts and money for a new roof is the one whose story goes round at Tihar. Many people really do well: in 2024 the money sent home by workers abroad came to about a quarter of the size of Nepal’s whole economy, one of the highest shares in the world. But the stories that travel home are mostly the ones people are glad to tell. A worker who came back poorer than he left, or ill, has no gift to show. And some don’t come back. In the sixteen years from 2008/09 to 2023/24, the Foreign Employment Board gave help to the families of 13,223 Nepali workers who had died abroad, more than 1,200 in each of the last two of those years. Many of the deaths were recorded only as cardiac arrest, heart attack or “natural causes”; road accidents, suicide and accidents at work took many others.

None of this means going abroad is a mistake. It means the stories at a family gathering can’t tell you your own chances. For that you’d need the whole crowd that left: how many got the job they were promised, how many paid off their loans, how many came home with less. Those numbers are harder to find than a cousin’s story, and they matter more.

Who isn’t in the picture

Once you start looking, the filter is everywhere. The businesses whose founders give talks are the ones that didn’t fail. The old buildings people admire are the ones still standing. The diets in the magazines come with photos of the people they worked for. In each case the useful question is the same: who isn’t in the picture, and why not?

It comes in three parts. How many set out? What happened to the rest? And would the ones you’re hearing about have done just as well anyway? You don’t always need exact answers. Even a rough guess, like the 9,000 students for 2,500 places, changes what a billboard seems to say.

A filter isn’t always a problem, though. It does harm only when what decides who gets heard is the very thing you want to know. If a hundred people who drank the tea were picked by drawing names from a hat, the happy and unhappy alike, their stories would speak fairly for everyone who drank it. Trouble starts when getting better is what makes a person speak up. Bomber crews didn’t come home because their holes were in the body of the plane. Holes in the body of the plane were what let them come home, and that’s why counting them pointed the wrong way.

There’s one last twist, and it’s about Wald himself. The way his story is usually told, the generals wanted to add armor where the returning planes had the most holes, and Wald told them to put it where there were none, around the engines, and his advice saved the bombers. Some of that is in the record, and some isn’t. W. Allen Wallis, who ran the group’s research, wrote in 1980, more than thirty-five years later, that the military “was inclined to provide protection for those parts that on returning planes showed the most hits”, and that Wald, assuming hits landed evenly over a plane, worked out how to read the ones that were missing. So the mistake in the story was real. But Wald’s memos are pages of equations and a made-up example, and they say nothing about what the military should do. Nobody has found a record of him telling the generals where to put the armor, or of anyone moving armor because of him. The pictures that go round online, of a plane covered in red dots, are drawings made long afterward. Wald’s method was used in that war and again in Korea and Vietnam, but the scene of the clever mathematician turning the generals round is, at best, a guess at what might have happened.

So the story of survivorship bias has a survivor of its own. The careful memos, full of symbols, were read by few people. The version with the generals was the better story, so that was the one people told, and so it was the one that reached you. Wald didn’t live to see any of it. He and his wife died in 1950, when their plane crashed in the Nilgiri hills of southern India, where he had gone at the Indian government’s invitation to lecture.

Next time a story reaches you polished and complete, the man at the center, the problem, the brilliant answer, it’s worth asking the same question of the story itself. Where did it first come from, and what do other people say about it?

Found a mistake? Please tell us.