Skip to content

← Thinking Clearly

Changing your mind by the right amount

Chapter 17 · Numbers without heavy math

In this chapter 4 sections
  1. Where the dial starts
  2. How strong a clue is
  3. Big claims need big evidence
  4. Forecasts, not predictions

On Sunday, 26 April 2015, the day after the great earthquake, a text message spread from phone to phone in parts of India. “North India will face next earthquake at 8.06 pm,” it said. It gave the size, 8.2, said the government wanted people out of their houses, and ended: “News from NASA. Plz forward message as much as u can.” The same week, in Kathmandu, people were passing on dates and times for the next big shock, from social media and from people the Kathmandu Post called “self-proclaimed” astrologers.

In the days that followed, a very different kind of message came from the United States Geological Survey (USGS), the American government’s earthquake scientists. It named no day and no hour. It gave chances: of a strong aftershock in the coming week, of a bigger one, of one bigger than the first earthquake. It sounded far less certain than the text message, and it was worth far more.

How much should a message like either one move you? The answer comes in two steps, and the rest of this chapter is those two steps. First, ask how often claims of this kind turn out true. That is where your dial starts, before you look at this particular claim at all. Then look at the evidence in front of you and ask how much more often you would see it if the claim were true than if it were false. That says how far to move. A clue you would see just as often either way shouldn’t move you at all. A clue that almost never turns up unless the claim is true can move you a long way.

That’s all there is to it. The idea goes back to an English minister more than 250 years ago, and the sums can get hard, but the thinking needs no formula.

Where the dial starts

Every claim arrives with a past. Before you know anything about this message, you already know something about messages like it, and that is the place to start.

So take the one from NASA. Of all the messages ever sent that named the hour of an earthquake, how many came true? The USGS answers plainly: neither it nor any other scientists have ever predicted a major earthquake, and nobody expects to in the foreseeable future. A real prediction would have to give the time, the place and the size, and none ever has. The European seismologists who track quakes around the world put it in capital letters on their page about Nepal’s earthquake: “EARTHQUAKE PREDICTION DOES NOT EXIST!” So a message that names 8.06 pm starts very near zero, and it starts there before you have read a word of it.

That can feel unfair to the message. Shouldn’t each claim be judged on its own? It is judged on its own, in the second step. But skipping the first step is how sensible people end up believing unlikely things. Something you’ve just been told feels like news, and the long record of claims just like it doesn’t come to mind at all.

The starting point isn’t a fixed number written in the sky. It’s your best judgment of how often things like this hold, and different people may pick somewhat different ones. That matters less than you’d think. If the evidence is strong, people who started in different places end up close together. If it’s weak, they stay roughly where they began, which is right, because weak evidence shouldn’t settle anything.

How strong a clue is

Now the evidence. Its strength isn’t how convincing it feels. It is a comparison of two numbers: how often you’d see this clue if the claim were true, and how often you’d see it if the claim were false.

Alan Turing, who worked on the German navy’s Enigma cipher at Bletchley Park during the Second World War, wrote this down for the codebreakers in a paper Britain kept secret until 2012. The example he gives for the rule has nothing to do with codes. Suppose, he wrote, that one man in five dies of heart failure. Of the men who die of heart failure, two in three die in their beds. Of the men who die of anything else, one in four does. A certain man died in his bed. How likely is it that his heart gave out? And then, in brackets: “My facts are no doubt hopelessly inaccurate.” They were only there to show the method.

Count it out with 60 men, the way the last chapter counted a thousand women. One in five, 12 of them, die of heart failure, and two thirds of those, 8, die in bed. The other 48 die of something else, and a quarter of them, 12, die in bed. So 20 men died in bed, and 8 of them had heart failure. Knowing he died in bed has moved the chance from 1 in 5 to 8 in 20, from 20 to 40 on the dial.

The bed is a clue of modest strength. Heart failure makes dying in bed nearly three times as likely as other causes do, so it nudges the dial without sending it anywhere dramatic. A clue that heart failure made a hundred times as likely would have carried the dial much further. A clue equally likely either way, such as the color of his shirt, wouldn’t have moved it at all.

Turing’s codebreakers had dozens of clues at a time, and he found a neat way to combine them. He gave each clue a score for its strength, so that clues could simply be added up as the evidence came in, a strong clue scoring many points and a weak one few. He called the unit a ban, after the town of Banbury, where the long sheets of card for his method were printed. A tenth of a ban was a deciban. I. J. Good, who worked beside Turing, later wrote that a deciban was about the smallest step in the weight of evidence that a person’s judgment can feel, as a decibel is about the smallest difference in loudness the ear can hear.

The rule underneath, which Turing called the factor principle, wasn’t his. It is named after Thomas Bayes, a Presbyterian minister in England. After Bayes died in 1761, his friend Richard Price found an essay among his papers, “which, in my opinion, has great merit,” and sent it to the Royal Society, where it was read in 1763. The French mathematician Pierre-Simon Laplace later did the most to develop it and spread it. It’s now called Bayes’s rule, and Turing’s paper calls it that too.

Turing added one condition, and it matters as much as the rest. Clues add up only if they are separate pieces of evidence. The bed and the man’s father having died of heart failure are two clues. The same text message from ten friends is one clue, not ten. Every forward of “News from NASA” rests on whoever wrote it first, and if that person knew nothing, a thousand forwards still know nothing.

So how strong a clue is the message itself? Ask the two questions. If NASA really could predict an earthquake, would the warning arrive as an unsigned text asking to be forwarded? Hardly. And if nobody can predict earthquakes, would such a text still go round? Plainly yes: it did. A seismologist at the India Meteorological Department said so at once: “It is fake. The NASA … cannot predict earthquakes.” The name NASA was a clue that fakes carry just as easily as real news, so it should hardly move the dial at all.

Try it

Nudge the Dial

Choose where a claim starts, then add clues. A ×10 clue turns up ten times as often if the claim is true as if it’s false (against it, the other way round).

25507501001%where it started
A clue for it:
Against it:
True, of 1,000 like it
10

No clues yet: the dial sits where claims like this usually stand. Add a clue for it or against it.

Start at 1 in 2 and a ×10 clue takes the dial to about 91. Start at 1 in 1,000 and the same clue takes it to 1. The clue is just as strong in both cases. What differs is how far it had to carry the claim.

Big claims need big evidence

That’s why the astronomer Carl Sagan’s line, “extraordinary claims require extraordinary evidence”, is good sense and not stubbornness. An extraordinary claim is one that starts near zero, and only very strong evidence, or a lot of separate evidence, can carry it far.

The best recent case came from physics. In September 2011 a team of scientists working on an experiment called OPERA announced something they couldn’t explain. They had sent tiny particles called neutrinos from CERN, near Geneva, through the rock of the Earth to a laboratory under the Gran Sasso mountain in Italy, about 730 kilometers away. By their measurements the neutrinos arrived about 60 billionths of a second sooner than light would have. Nothing is supposed to travel faster than light. Einstein’s relativity forbids it, and relativity had passed test after test for over a century.

This was not a careless claim. The team had spent six months checking before they announced it, and a few weeks later they repeated the measurement in a different way, with the same answer. They wrote that they deliberately weren’t offering any explanation, and asked other physicists to go over their work.

Most physicists stayed doubtful, and they weren’t being closed-minded. The measurement was strong evidence that the neutrinos had arrived early by the team’s clocks. But the dial they had to move wasn’t “are the clocks right?”. It was “is relativity wrong?”, and that claim started very low. Against it stood a far likelier story: that somewhere in a timing chain of satellite receivers, cables and electronics stretching from Switzerland to a cavern in Italy, something was off by a few billionths of a second. That story would produce exactly the same measurement. A clue that both stories predict equally well can’t tell them apart.

In early 2012 the team found two faults in its own equipment. The bigger one was a fiber-optic cable that hadn’t been screwed in properly. The delay it caused had made the neutrinos look faster than they were. In June 2012 four experiments at Gran Sasso reported that neutrinos travel at the speed of light, as far as anyone can measure. CERN’s research director, Sergio Bertolucci, said the result “isn’t as exciting as some would have liked,” but “it is what we all expected deep down.”

None of this means surprising claims are always false. Some turn out true, and Turing’s own paper shows how they win. Suppose, he wrote, three messages in a new kind of code all have the letters V and G in the 17th and 18th places. Is that a rule of the code? He guessed a rule like that was very unlikely, about one chance in five million to begin with. But three matches by luck would be far rarer still, about one in three hundred million, so the evidence was strong enough to carry the rule to odds of about 60 to 1 in its favor. A big claim can beat a tiny start. It needs evidence that is far more likely if it’s true than if it isn’t, and evidence that rules out the dull explanation, which in physics usually means other teams with other equipment getting the same result. The neutrinos never got that. Turing, even at 60 to 1, added a warning: other explanations he hadn’t thought of might fit as well, and it was always necessary to bear them in mind.

It works the other way too. A dial set to exactly 0 or 100 can never move, because nothing multiplies zero into anything else, and no amount of evidence ever takes a dial all the way there. Richard Price saw this in the appendix he wrote for Bayes’s essay. Imagine someone brought into the world for the first time, he wrote, who watches the sun set and has no idea whether it will come back. Each sunrise makes the next one more expected, but “no finite number of returns would be sufficient to produce absolute or physical certainty”. Even a million sunrises leave a little room. For the sun that room is too small to matter. For most beliefs it’s what lets the right evidence get in.

Forecasts, not predictions

Which brings the story back to Nepal in the spring of 2015, where the two kinds of message were going round together.

A row of dark canvas tents on a dry, grassy open ground, with a few trees and tall buildings behind them in the haze.
Figure 17.1 Tents on an open ground in central Kathmandu on 2 May 2015, a week after the earthquake. The photographer didn’t say who was living in them. Photo: mohigan, CC BY-SA 3.0

A prediction says what will happen and when: an earthquake at 8.06 pm. A forecast says how likely something is over a stretch of time. The USGS’s forecasts came from how aftershocks had died away after big earthquakes in the past, fitted to the aftershocks recorded in Nepal since 25 April. A table said to come from the USGS went round in early May. It gave about 1 chance in 100 of a quake bigger than magnitude 7 between 1 and 31 May.

On 12 May, a magnitude 7.3 aftershock struck.

So were the people who feared another big earthquake right after all, and the forecast wrong? No to both, and the reason is the heart of this chapter. People were right to fear a strong aftershock, because one really could come, and a forecast told them so in numbers. But nobody who named a day knew anything about that day. A chance of 1 in 100 is supposed to come true about once in a hundred such months, so one strong aftershock can’t show that the forecast was wrong. The USGS said afterward that the 7.3 had been a low-probability event for that week, unusual but not unheard of, and part of the same sequence.

And then the forecast did what a forecast should: it changed by the right amount. On 13 May the USGS raised its chances, because a big new aftershock brings aftershocks of its own. For the week ahead it gave about a 15% chance of a magnitude 6 to 7 aftershock and about 1 in 100 of one from 7 to 7.8. A week later, as the days passed, it lowered them again: 6%, and 1 in 200. Each time it said there was “no way to predict the exact date or time of an earthquake or aftershock”, and each time it warned people to take care in and around damaged buildings. The messages with dates never moved at all. The 8.06 pm came and went without the 8.2.

That is the difference worth carrying away. A forecaster starts from what usually happens, moves as far as each new piece of evidence deserves, and keeps moving. Someone who predicts a date starts from nothing, moves no matter what, and never comes back to check. The forecast sounded less sure, and it was the one telling the truth about how much anyone knew.

Most of the evidence that moves the dial furthest in everyday life comes from one kind of question: does this make that happen? Does a herb lower blood sugar, does the vaccine stop typhoid, does the rain bring the landslide? That is the hardest kind of claim to show, and the most worth learning to judge.

Found a mistake? Please tell us.