Can 1,000 People Speak for 3 Crore?

Think of a big pot of dal. To find out whether it needs more salt you don’t eat the whole pot: you stir it and taste one spoonful. One spoonful tells you about a small pot or a huge one equally well, as long as the pot is well stirred. If you didn’t stir, and all the salt sank to the bottom, a spoonful from the top would mislead you however big the spoon.

A survey is a spoonful of people. The words statisticians use:

• The population is everyone you want to know about: all adults in Nepal, every bag of cement from a factory this month, every fish in Phewa Lake.
• A sample is the part you actually look at: the 1,000 people asked, the 50 bags tested.
• The number you want, such as the share of all adults who want a Baisakh start, is a parameter of the population. You almost never know it. The same share worked out from the sample is a statistic, and it is your estimate of the parameter.

Stirring the pot, for people, means choosing at random. In a simple random sample every person in the population has the same chance of being chosen, and so does every possible group of 1,000. Picture every adult’s name on a slip in one enormous drum, turned over and over, and 1,000 slips drawn out. Nobody chooses who is asked: not the surveyor, not the people themselves. Then each person drawn is like a fresh toss of a weighted coin: if 40% of the country would say yes, each slip has a 40% chance of being a “yes”, whether the drum holds 2,000 slips or 3 crore. Unless the sample empties a good part of the drum, its size hardly enters.

The widget below is a made-up country where exactly 40% of the people would say yes. Each survey asks people chosen at random, and its share of yeses adds height to that share’s bar on the chart.

Things to notice:

1. Every survey is a little wrong. Hardly any lands exactly on 40%. That is not a mistake by the surveyor: it is the luck of the draw, called sampling error, and no survey can escape it.
2. Bigger samples are less wrong. With 10 people, the middle 95% of results runs from 10% to 70% (in 4,000 surveys we ran). With 100 people, the middle 95% runs from about 30% to 50%. With 1,000, from about 37% to 43%.
3. The size of the country doesn’t matter. With 1,000 people, a country of 1 lakh and a country of 3 crore give the same spread. Only the country of 2,000 is different: there you are asking half of everyone, and the results crowd closer to 40% (about 38% to 42%).

So your uncle’s one in 29,000 is the wrong thing to look at. What decides how sure you can be is the number of people asked, 1,000, and nothing else (as long as it is a small part of the population).

From 10 to 100 people, ten times as many, the spread shrank from about ±30 points to about ±10. From 100 to 1,000, another ten times, it shrank to about ±3. Each time the spread got about 3.16 times smaller, and $3.16 \approx \sqrt{10}$. The spread shrinks with the square root of the sample size. Here is why, using nothing more than Grade 10 probability.

One person’s answer

Write a “yes” as 1 and a “no” as 0. If the true share is 40%, one person drawn at random is a 1 with probability 0.4 and a 0 with probability 0.6. On average the answer is 0.4, but it is never actually 0.4: it is off by +0.6 or −0.4. To measure the typical error, square the errors and average them, weighted by their chances:

$$0.4 \times 0.6^2 + 0.6 \times (-0.4)^2 = 0.144 + 0.096 = 0.24.$$

In general this is p(1 − p) for a true share p: 0.24 here. Its square root, about 0.49, is the typical size of one person’s error. It is the standard deviation of Grade 10 Optional Math.

The errors of many people add up slowly

Now add up the answers of n people: the count of yeses. Its error is the sum of their errors, and those partly cancel, some +0.6 and some −0.4.

Look at two people. Their total error squared is

$$(e_1 + e_2)^2 = e_1^2 + e_2^2 + 2e_1e_2.$$

Average this over all four ways two people can answer. The first two terms each average 0.24. The last, 2e₁e₂, averages zero: because the two are drawn independently (exactly so if each slip goes back into the drum before the next draw, very nearly so if not; see below), person 2 is just as likely to err up or down whatever person 1 did, so the products e₁e₂ cancel (by the multiplication law: 0.4 × 0.4 × 0.36 + 2 × 0.4 × 0.6 × (−0.24) + 0.6 × 0.6 × 0.16 = 0). So the squared errors simply add: 0.24 + 0.24.

The same goes for any number of people: the count’s average squared error is n × 0.24, so its typical error is $\sqrt{n \times 0.24}$, growing only like $\sqrt{n}$. The share is the count divided by n, so its typical error is

$$\frac{\sqrt{n \times p(1-p)}}{n} = \sqrt{\frac{p(1-p)}{n}}.$$

This is called the standard error of the share. For 1,000 people and p = 0.4, it is $\sqrt{0.24 / 1000} \approx 0.0155$: 1.55 percentage points. Ten times more people make it $\sqrt{10}$ times smaller, as the widget showed.

The population’s size appears nowhere in the formula. It enters only when the drum noticeably runs out: after many yeses, a no becomes likelier, and the errors cancel even better. The standard error is then multiplied by $\sqrt{(N - n)/(N - 1)}$ for a population of N. For 1,000 out of 2,000 that is about 0.71, which is why the small country’s results crowded closer; for 1,000 out of 3 crore it is 0.99998, which is 1 for every purpose.

From standard error to margin of error

Many surveys’ results pile up in the hump you saw in the widget. With samples of a few hundred or more, it is very close to the normal distribution, the bell curve first found as an approximation by Abraham de Moivre in 1733 for exactly this question, the number of heads in many tosses. For a bell curve, 95% of the results fall within 1.96 standard errors of the middle.

So a survey reports its margin of error as 1.96 standard errors:

$$\text{margin of error} = 1.96\sqrt{\frac{p(1-p)}{n}}.$$

The surveyor doesn’t know p, so uses the share the survey found, or plays safe with p = 0.5, where p(1 − p) is largest. With p = 0.5 the formula becomes $1.96 \times 0.5 / \sqrt{n} = 0.98 / \sqrt{n}$, or very nearly $1/\sqrt{n}$: a handy rule.

People asked	Margin of error (95%, at p = 0.5)
100	±9.8 points
400	±4.9 points
1,000	±3.1 points
2,500	±2.0 points
10,000	±1.0 point
1,00,000	±0.3 points

There is the TV news’s “±3 points” for 1,000 people. And there is the price of precision: to halve the margin you must ask four times as many people. Going from ±3 to ±1 costs ten times as much fieldwork, which is why so many polls settle on about a thousand.

The margin of error is easy to misread. It doesn’t mean the survey is 95% correct. It is a promise about the method: if you did surveys like this over and over, about 95% of them would land within their margin of the truth. Each survey’s result with its margin either side, from 59% to 65% in the news story, is called a 95% confidence interval.

The widget below runs twenty surveys of 1,000 people at once, each with its own interval, in the country where the truth is 40%.

About one survey in twenty misses (in 20,000 simulated surveys, 94.6% of the intervals caught 40%). The surveyor of a red one did nothing wrong: they were unlucky, and will never know it, because in real life nobody sees the dashed line. With half the margin, the intervals look more impressive and are wrong about a third of the time.

Everything so far assumed a stirred pot: a random sample. Now a famous poll that wasn’t.

In 1936 the United States chose between President Franklin Roosevelt and Alf Landon. The Literary Digest, a magazine whose polls had picked every winner since 1916, mailed questionnaires to ten million people and got back 2.38 million. Its forecast: Landon would win easily, with 57% of the vote. On election day Roosevelt won 60.8% and carried 46 of the 48 states; Landon got 36.5%.

With 2.38 million answers, the margin of error of a random sample would be about ±0.06 points. The Digest’s share for Landon was out by more than 20. Its trouble was who was in it. Its lists were its own readers, car owners and telephone users: people with money to spare in the Great Depression, who leaned toward Landon. Worse, those who sent the card back were mostly the ones who disliked Roosevelt strongly. Later research found this second problem, non-response, did the most damage.

The same year a young pollster, George Gallup, asked about 50,000 people, chosen so that each kind of voter appeared in the sample in proportion to its share of the country. He forecast that Roosevelt would win with 55.7%. That too was five points off, but it named the winner, and polling never looked back.

A systematic lean of this kind is called bias. Sampling error goes down as the sample grows; bias doesn’t go down at all. Try it with the widget, now in a country of 3 crore where a third of the people live in cities and 70% of them say yes, against 25% of everyone else: 40% in all. In the online poll, a city person is five times as likely to answer.

The online poll’s results gather round 57%, because five in every seven of its answers come from the cities. With a lakh of answers, nearly every poll comes out between about 56.8% and 57.4%: wonderfully consistent, and 17 points wrong. Precision is not accuracy. A margin of error measures only the luck of a random draw; it says nothing about bias, and a poll that quotes one for a sample that wasn’t random is quoting a number that doesn’t apply.

Where bias comes from

• Selection bias: the way people are reached leaves some out. An online poll misses people with no internet, a survey on Kathmandu’s New Road everyone who doesn’t shop there. A convenience sample (whoever is easiest to reach: your class, the first 100 people at the bus park) is the worst offender.
• Non-response bias: the people chosen don’t all answer, and those who do are different. A survey of 1,000 households in which 400 refused is no longer a random sample of anything.
• Self-selection: a phone-in vote or a “like if you agree” post counts only those keen enough to take part.
• Response bias: the question itself pushes people. “Do you agree that our hard-working teachers deserve better pay?” gets more yeses than “Should teachers’ pay go up, even if school fees rise?” People also shade the truth on questions that embarrass them.

None of these shows in the margin of error, and none gets better with a bigger sample.

A simple random sample of a whole country is hard: there is no drum of every adult’s name, and a random 1,000 people would be scattered over hundreds of wards. Real surveys use other designs, still built on chance:

• Systematic sampling: start at a random place among a list’s first k entries and take every k-th after it, such as every 50th household on a ward’s register. Easy, unless the list repeats a pattern every k.
• Stratified sampling: split the population into groups (strata) that differ, such as provinces, or city and village, and draw a random sample in each, in proportion to its size or deliberately more in small groups so they can be reported on too. With each group’s sample in proportion to its size (and groups of any real size), it is no less precise than a simple random sample of the same size, and usually more.
• Cluster sampling: choose whole groups at random (wards, villages, school classes) and then ask people within them. Much cheaper to visit; a little less precise, because neighbors tend to be alike.
• Quota sampling: fill fixed numbers of each kind (so many women, so many farmers) with whoever the interviewer finds. It looks like stratifying, but the interviewer, not chance, picks the people within each kind (the friendliest, the nearest), so bias creeps in, and since chance didn’t choose, no margin of error really applies. Gallup and his rivals polled this way into the 1940s; in 1948 all three big polls wrongly picked Thomas Dewey to beat Harry Truman, and pollsters turned to chance.

Surveys also measure averages: the mean size of a household, the mean height of Grade 9 students. The reasoning is the same. One household is off from the mean by a typical amount, its standard deviation s; the errors of n households drawn at random partly cancel, so the sample mean’s typical error is

$$\text{standard error of the mean} = \frac{s}{\sqrt{n}},$$

and its 95% margin of error is $1.96\,s/\sqrt{n}$. (For a yes/no answer, $s = \sqrt{p(1-p)}$, and this is the formula from before.)

Check the data first: a household typed in as 44 people instead of 4 is an outlier that is an error, and moves the mean of 400 noticeably. Other outliers are real (some households are large) and stay.

The theory under those surveys took shape at the same time. In 1934 Jerzy Neyman, a Polish statistician then lecturing at University College London, compared random, stratified samples with choosing “typical” places by judgment, in the paper that began modern scientific sampling; it also held the first form of the confidence interval, whose full theory he gave in 1937.

• Official statistics. Nepal’s census counts everyone every ten years; between censuses, sample surveys such as the Demographic and Health Survey take the country’s measure. India’s National Sample Survey has measured poverty, employment and spending since 1950.
• Opinion polls, business research and medical trials.
• Quality control. A factory can’t crush every bag of cement; it tests a random sample, and buyers accept or reject a whole batch from one (acceptance sampling).

• Does Ice Cream Cause Drowning? Correlation, causation and the fair test, which uses a coin for the same reason a survey does.
• Can a Treatment Win in Every Group but Lose Overall? Simpson’s paradox: why the groups in a sample matter.
• Why Do So Many Numbers Start with 1? Benford’s law, and how it catches made-up data.
• What Does an Average Hide? (Grades 6–8) Means, medians and the spread an average leaves out.
• How Can Throwing Rice Find π? (Computing, Going Further 6–8) Random samples of points that estimate a number, and how the estimate improves with more.
• Grade 10 Math: Probability, the multiplication law under the square root, and Grade 10 Optional Math: Statistics, the standard deviation.