Skip to content

← Thinking Clearly

Who was asked

Chapter 14 · Numbers without heavy math

In this chapter 5 sections
  1. A spoonful of dal
  2. Millions of the wrong people
  3. A quarter of a million a week
  4. Nepal’s spoonful
  5. What was asked, and how

In the autumn of 1936 an American magazine called the Literary Digest set out to find who would be the next president. Franklin Roosevelt wanted a second term, and a Republican, Alf Landon, wanted to stop him. The Digest didn’t guess. It asked. It mailed out ten million ballots, postcards that people could mark and send back, to names taken from telephone books, lists of car owners, club rosters and other directories. Every day its staff addressed more than a quarter of a million envelopes by hand.

The magazine had done this before, and it had picked the winner in each of the last four presidential elections. It was so sure of its method that it told its readers, as the 1936 poll began, that it was being run “in precisely the same manner as the many others which have proved nearly 100 per cent. correct every time.” Almost 2.4 million ballots came back. The Digest counted them, state by state, and announced that Landon would win, with 370 electoral votes against Roosevelt’s 161.

Roosevelt won easily. He took more than 60% of the vote and every state except Maine and Vermont. With well over two million answers, the Digest was wrong by about twenty points. Within two years the magazine was out of business.

Meanwhile a pollster named George Gallup, who had set up his own polling institute the year before, asked far fewer people and said Roosevelt would win. He was off too, by about seven points, but he was on the right side.

How can a much smaller poll beat 2.4 million answers? Because what matters most about a poll isn’t how many people were asked. It’s which ones.

A black-and-white portrait of a smiling, balding man in a striped tweed jacket, white shirt and patterned tie.
Figure 14.1 George Gallup, in a press photo from the Toronto Star’s archive. Photo: Norman James, Public domain

A spoonful of dal

Nobody can ask everyone. Even a country’s census, which tries, comes only once every ten years. So when someone wants to know what a whole country thinks, or how many families own a phone, or how many children have been vaccinated, they ask some people and use the answers to stand for all of them. The people asked are called a sample.

It works the way tasting works. To find out whether a pot of dal needs salt, you don’t eat the whole pot. You stir it and taste a spoonful. One spoonful is enough, and it’s enough whether the pot holds two plates or two hundred, as long as it was stirred. The stirring is what matters. Taste from the top of an unstirred pot, where the salt hasn’t reached, and a bigger spoon won’t help you.

For a poll, the stirring is chance. If everyone in the country has the same chance of being picked, and chance alone decides who is, then the people picked will be like everyone else on average: as many old and young, rich and poor, town and village, for and against, as in the country itself, give or take a little luck. Such a sample is called random. It doesn’t mean careless. It means chosen by a lottery that nobody can tilt.

A random sample can still go wrong in three ways, and every bad poll is some mix of them. It can be too small, so luck has too much say. It can be drawn from the wrong list, so some people never had a chance of being picked. And the people picked can choose for themselves whether to answer, so the ones who reply aren’t like the ones who don’t. A fourth trap sits beside all three: the question itself.

The first is the easiest to measure. With a random sample of about a thousand people, a share like “what part of the country says yes” usually lands within about 3 percentage points of the truth, as far as the luck of the draw goes. Statisticians call that the margin of error, and “usually” has a precise meaning here: 19 times out of 20. Ask a quarter as many and the margin doubles. Ask four times as many and it halves. And here is the surprise that makes polling possible: as long as the sample is a small part of the whole, the size of the country hardly matters. A random thousand tells you about as much about Nepal’s 29 million people as about a district of a few hundred thousand. It is the spoonful that counts, not the size of the pot.

That margin is easy to forget when a poll is cut into pieces. A survey of a thousand people may say something about women under 25 in Karnali, but if only forty such women were in it, the margin for them is wide enough to drive a bus through. The headline number may be solid while the one underneath it is close to guesswork.

Millions of the wrong people

For decades the usual explanation of the Digest’s mistake was the second kind of error, the wrong list. In 1936 America was in the middle of the Great Depression. People who had a telephone or a car were better off than most, and better-off people, the story went, leaned toward Landon. So the Digest had polled the comfortable and missed the poor. Gallup himself told it this way for the rest of his life, and it’s in many textbooks still.

The evidence, when people finally looked, points somewhere else as well. In 1937 one of Gallup’s own surveys asked people whether they had received a Digest ballot the year before, whether they had sent it back, and whom they had voted for. Among people who had received a ballot, Roosevelt’s supporters were in the majority. The list wasn’t the main problem. The problem was who mailed the card back. Landon’s supporters, it seems, were keener to be heard, and returned their ballots far more often than Roosevelt’s did. Only about one ballot in four came back, and those that did were tilted. If everyone who got a ballot had returned it, the Digest would probably have picked the winner. One careful reanalysis of that survey puts about two thirds of the Digest’s error down to who answered, and a third to the list.

The evidence is thin, since it rests on people remembering a postcard a year later, and scholars still weigh the two causes differently. But the lesson is the same either way. A poll can go wrong before a single question is asked, in the choice of whom to ask, and again after, in who bothers to answer. A sample whose answers come from people who chose to give them speaks for people who like to give answers. Those are often the people who feel most strongly, and that is rarely everyone.

Size did nothing to help. If anything, it hurt. Nearly 2.4 million answers gave the Digest a number that looked rock solid, and every extra card only repeated the same tilt with more confidence. A bigger spoon from the top of an unstirred pot just tells you, very precisely, what the top tastes like.

A quarter of a million a week

It would be nice to think that this lesson was learned in 1936. It was learned again in 2021, with numbers far bigger than the Digest’s.

That year, as COVID-19 vaccines reached the United States, everyone wanted to know how many adults had had their first dose. Several surveys asked. One, run by researchers at Carnegie Mellon University with Facebook, invited Facebook users to answer questions and got about 250,000 replies every week. Another, by the United States Census Bureau, got about 75,000 replies at a time. A third, by the polling companies Axios and Ipsos, asked only about a thousand people each time.

The unusual thing is that for once the true answer was known. Every dose given was counted by the government’s Centers for Disease Control and Prevention, so each survey could be checked against the real figure. A team of statisticians from Oxford, Harvard and Stanford did exactly that.

Official count of doses givenFacebook survey, ~250,000 a weekIpsos poll, ~1,000 each time0%20%40%60%80%17 pointstoo highJanFebMarAprMayUS adults with at least one dose, 2021
Figure 14.2 How many American adults had had at least one vaccine dose, January to May 2021: the official count, the Facebook survey and the small Ipsos poll. The bars on the Ipsos dots show its margin of error. The Facebook survey’s is too thin to see.

The thousand-person poll followed the official count closely all the way. The big surveys didn’t. By mid-May the Facebook survey put the share of vaccinated adults about 17 points higher than it really was, and the Census survey about 14 points higher. With so many answers, each big survey’s margin of error was tiny, so each was very precise and very wrong, the Digest’s mistake once more. By April, the statisticians worked out, a week of the Facebook survey’s quarter of a million answers told you no more than ten people picked at random would have.

What went wrong was who answered. Not everyone uses Facebook, and of those who do, the ones who stop to fill in a health survey aren’t typical. People with a college degree were heavily overrepresented among the answers, and they were also more likely to have been vaccinated. The Facebook survey adjusted its numbers for age and sex, but not for education, so that tilt stayed in. It can’t be the whole story, though: the Census survey did adjust for education and was still far too high. Whatever made people answer, it went along with having been vaccinated, in ways that no adjustment for age, sex or schooling could take out. The survey’s own team had said that its numbers were best for comparing places and weeks, and that its totals should be read with caution, which was fair. But a number that big, with a margin of error that small, is hard not to believe.

The Ipsos poll did something slower and duller. Its people came from a standing panel that Ipsos had built by choosing homes by chance from the country’s list of postal addresses and writing to them. People who agreed to take part but had no internet at home were given a tablet and a connection, so that the poll didn’t quietly become a poll of people who are online. That care, not the number of people, is what made a thousand beat a quarter of a million.

Nepal’s spoonful

Nepal has its own version of the careful thousand, only bigger. About every five years since 1996 the Nepal Demographic and Health Survey asks families about their children’s health, births, vaccinations, nutrition and much else, and governments and health workers plan around its numbers. The sixth went to the field from January to June 2022.

Its list was the census: every one of the 36,020 small areas, wards as they were in 2011, that the census had mapped. From that list chance picked 476 areas, giving bigger ones a bigger chance. In each picked area, teams first made a list of every household. Then chance picked thirty households from that list, 14,280 in all, in every province. The interviewers were told to visit only those houses. If nobody was home, they couldn’t knock next door instead, because next door is exactly how a sample drifts toward the people who happen to be easy to find. They were to come back, at least three times.

Of the 13,833 picked houses where someone lived, 13,786 were interviewed. That is more than 99%, against the Digest’s one in four. So when the survey says what share of Nepal’s children are fully vaccinated, it is fourteen thousand households speaking for a country of 29 million people, and the stirring is why that can be trusted.

Compare that with a poll on a news site’s Facebook page, the kind that asks its followers to tap Yes or No and gets fifty thousand taps by evening. Those fifty thousand chose themselves twice. They follow that page, which already makes them a particular kind of person, and they cared enough to tap. Such a poll can tell you what the page’s keenest followers feel. It can’t tell you what Nepal thinks, however many taps it gets.

What was asked, and how

Even a perfect sample can’t rescue a bad question, because people answer the words in front of them. In January 2003 the Pew Research Center asked Americans whether they favored taking military action in Iraq to end Saddam Hussein’s rule. 68% said yes. When the same question added “even if it meant that U.S. forces might suffer thousands of casualties,” only 43% said yes, and more said no. They were answers to two different questions.

So the words matter as much as the people. A question that names only one side of a choice, or comes after another question that sets a mood, or uses a word that sounds kind or harsh, can move the answers by many points before anyone has picked a sample at all. Good pollsters print their exact questions so that others can judge them. A number reported without its question is only half a number.

Statisticians have a habit that sums up the whole chapter. Before they look at any data, they ask what the question really is, and then how they would go about answering it: whom to ask, how to choose them, how to word it, what would count as an answer. Only then do they collect the data, analyze it and draw conclusions. Statistics teachers in many countries teach it as a cycle with an ugly name, PPDAC, for Problem, Plan, Data, Analysis, Conclusion, where the conclusion often raises the next problem and the cycle goes round again. The point is in the order. Most of the ways a number goes wrong are settled in the first two steps, long before anyone does a sum.

The same order works for reading a number someone else made. “72% of students prefer…” Prefer what, asked how? Which students, picked how, and how many of the picked ones answered? When you know that, you know what the number can carry. When you can’t find out, the number hasn’t earned much of your belief, however many people it claims to have asked.

Even the best sample, though, has the luck of the draw built in. Pick a thousand people fairly, then pick another thousand the same way, and the two answers won’t match exactly. That wobble is chance, and chance behaves in ways that surprise nearly everyone, statisticians included.

Found a mistake? Please tell us.