Data, Evidence and Decisions

Before an election, a newspaper asks 2,000 voters whom they will vote for, and then says what a country of millions will do. A ward office asks 40 households about their internet, and plans for the whole ward. A factory tests 10 bottles of every thousand it fills. In each case a few stand for many. When can they?

The whole group you want to know about is the population: every voter, every household in the ward, every bottle. The ones you actually ask or measure are a sample. (You met samples in Grade 7, in Telling what you found.) A number worked out from a sample, such as “35% of the households we asked have internet”, is an estimate of the population’s true number, which you usually never learn.

An estimate can miss the truth in two quite different ways.

Chance: sampling error

Ask 40 households chosen completely at random, and you might, by luck, get more with internet than the ward really has, or fewer. Ask another 40, and you get a different answer. The difference between an estimate and the truth that comes only from which members happened to be chosen is called sampling error. It isn’t a mistake anyone made; it is chance.

Sampling error shrinks as the sample grows, but slowly. In Grade 11 you saw that the error of a Monte Carlo estimate shrinks in proportion to 1 ÷ √(the number of trials) Grade 11: More trials, a steadier answer. A random sample is exactly such an estimate, so the same law holds: four times the sample halves the error, and a hundred times the sample cuts it to a tenth.

That is why 2,000 people can stand for a whole country. What matters is the size of the sample, not the size of the population: a random sample of 2,000 is about as good in a country of 30 million as in a city of 3 million, just as a spoonful of well-stirred dal tells you how salty the whole pot is, whether the pot is small or huge.

Bias: the wrong people

A sample is random when every member of the population has the same chance of being chosen, and chance alone decides who is. Then the people chosen are, on average, like everyone else. Any other way of choosing can tilt the sample towards one kind of member. When the way a sample is chosen makes its estimates miss the truth in the same direction, again and again, the sample is biased. The common ways:

• Easy to reach. Interviewers who knock only on the doors along the road, where the bus drops them, miss the households up the hill, which are often poorer and less connected. A sample of whoever is easiest to reach is a convenience sample.
• Chose themselves. An online poll on a news site is answered by people who read that site, have internet, and feel strongly enough to click. A self-selected sample speaks only for people like them.
• Didn’t answer. If people with no phone, or who are at work all day, never answer a phone survey, the survey hears from everyone else. This is non-response.
• Left off the list. A sample can only be chosen from a list of the population (a voter list, the households in a census, the numbers in a phone book). Whoever is missing from the list can never be chosen.

Bias doesn’t shrink when the sample grows. A million answers from the wrong people are as wrong as a thousand, only more confidently so.

An estimate on its own is only half an answer. “35% of households have a computer” from 20 households and from 20,000 households are very different claims. A careful report gives every estimate with a range that says how far it could be from the truth because of chance.

How big is that range? The honest way to find out would be to take many samples, as the widget above did, and see how much their answers vary. But a real survey costs money and weeks: you get one sample. In 1979 the American statistician Bradley Efron published a way to learn the spread from that one sample.

Resampling: the bootstrap

Here is a sample. A ward office asked 40 households, chosen at random, how much they spend on mobile data in a month. (Made-up numbers of a realistic size.)

import randomimport statistics spend = [200, 400, 800, 600, 400, 250, 550, 450, 400, 650, 1550, 1450, 850, 1050,         200, 1200, 450, 650, 650, 450, 300, 450, 300, 1000, 550, 200, 900, 350,         400, 600, 700, 300, 300, 650, 500, 900, 950, 600, 750, 600] print("Households:", len(spend))print("Average: Rs.", statistics.mean(spend)) random.seed(1)means = []for i in range(1000):    again = random.choices(spend, k=len(spend))   # 40 picks, with replacement    means.append(statistics.mean(again)) means.sort()low = means[25]      # 25 of the 1,000 are below thishigh = means[974]    # and 25 are above thisprint(f"Resampled averages: from Rs. {means[0]:.0f} to Rs. {means[-1]:.0f}")print(f"The middle 95%: Rs. {low:.0f} to Rs. {high:.0f}")

Households: 40Average: Rs. 612.5Resampled averages: from Rs. 459 to Rs. 770The middle 95%: Rs. 518 to Rs. 710

The idea: the best picture you have of the whole ward is the sample itself. So pretend the ward is made of these 40 households, copied over and over, and take new samples of 40 from it. random.choices(spend, k=40) picks 40 values from the list with replacement: each pick puts its value back, so one household can be picked twice and another not at all, just as a new random sample from the ward would give different households. Each such made-up sample is a resample, and taking many is called resampling, or the bootstrap.

The resampled averages vary, as real samples’ averages would. Sort them, cut off the lowest 2.5% and the highest 2.5%, and what is left, the middle 95%, is a 95% interval (statisticians also call it a confidence interval): the range of values the true average could reasonably have, given this sample. So the honest report is:

The households spend about Rs. 612 a month on mobile data (95% interval Rs. 518 to Rs. 710, from 40 households chosen at random).

The same works for a share. 13 of the 40 households have a computer at home: 32.5%. Put a 1 for each household with one and a 0 for each without, and resample:

import random answers = [1] * 13 + [0] * 27        # 13 of 40 households have a computerprint(f"Share: {sum(answers) / len(answers):.1%}") random.seed(2)shares = sorted(sum(random.choices(answers, k=40)) / 40 for i in range(1000))print(f"The middle 95%: {shares[25]:.1%} to {shares[974]:.1%}")

Share: 32.5%The middle 95%: 20.0% to 47.5%

20% to 47.5%: from 40 households, the ward could have anything from one household in five to almost one in two with a computer. That is worth knowing before the office spends money on the answer.

Climb from Kathmandu to Nagarkot and the air gets cooler. Go down to Chitwan and it gets hot. Everyone in Nepal knows that higher means colder. But how much colder, for each 1,000 m? And how sure can you be? A question like this, about how one number changes with another, is answered by fitting a line.

The data: Nepal’s weather stations

Nepal’s Department of Hydrology and Meteorology runs weather stations all over the country. For 15 of them, those whose records it sends to the World Meteorological Organization, the averages over the 30 years from 1991 to 2020, called climate normals, are published openly. Here they are: each station’s height in meters, its average temperature over the whole year in °C, and its rain in an average year in mm.

station,height,temperature,rainBiratnagar,72,24.9,1757Bhairahawa,108,24.9,1633Simara,137,24.4,1807Nepalgunj,165,24.6,1476Dhangadhi,184,24.2,1851Dipayal,563,23.2,1055Dang,663,22.2,1581Surkhet,720,22.1,1562Pokhara,827,21.5,3866Dhankuta,1192,19.9,937Kathmandu,1337,19.3,1503Okhaldhunga,1731,17.5,1742Taplejung,1744,16.8,1916Dadeldhura,1879,16.5,1345Jumla,2363,13.1,804

(Source: WMO Climatological Standard Normals 1991–2020, Nepal, sent by Nepal’s Department of Hydrology and Meteorology and published by the US National Centers for Environmental Information, which make them free for anyone to use.)

As Grade 10 taught, draw the data before anything else: a scatter chart, height across and temperature up (the widget below draws it, and the program at the end of this unit draws it in Python). The dots fall almost on a straight line, sloping down. A straight line can be described by two numbers: where it starts and how steeply it falls. Which line is the best one?

Misses and squares

Draw any line through the dots. For each station, the line gives a prediction: the temperature the line says a station at that height should have. The station’s real temperature minus the prediction is the line’s miss at that station, called its residual. Above the line, a residual is positive; below it, negative.

A good line has small residuals. As with the momo bills in Grade 11 Grade 11: One weight: the price of a plate of momos, square each miss, so that misses above and below both count and a big miss counts a lot, and add them up. The line with the smallest sum of squared residuals is the least-squares line, and finding it is called linear regression by least squares.

Unlike the momo price, which gradient descent found step by step, the least-squares line has an exact answer. With x the heights, y the temperatures, and x̄ and ȳ their means:

$$\text{slope} = \frac{\sum (x - \bar{x})(y - \bar{y})}{\sum (x - \bar{x})^2} \qquad \text{intercept} = \bar{y} - \text{slope} \times \bar{x}$$

You don’t need to learn why (it comes from calculus, or from Grade 11’s bowl, which has exactly one lowest point). You do need to read it. The bottom measures how spread out the heights are. The top adds up, station by station, “how far above the mean height” times “how far above the mean temperature”: a station that is higher than average and colder than average adds a negative amount, so when high stations are cold, the top, and the slope, are negative. And the second formula says that the line always passes through the point (mean height, mean temperature).

In Python, with no library:

import csv heights, temps = [], []with open("stations.csv") as f:    for row in csv.DictReader(f):        heights.append(int(row["height"]))        temps.append(float(row["temperature"])) n = len(heights)mx = sum(heights) / n                  # the mean heightmy = sum(temps) / n                    # the mean temperaturetop = sum((x - mx) * (y - my) for x, y in zip(heights, temps))bottom = sum((x - mx) ** 2 for x in heights)slope = top / bottomintercept = my - slope * mx print(f"{n} stations, mean height {mx:.0f} m, mean temperature {my:.2f} °C")print(f"slope: {slope:.5f} °C per meter, {slope * 1000:.2f} °C per 1,000 m")print(f"intercept: {intercept:.2f} °C")

15 stations, mean height 912 m, mean temperature 21.01 °Cslope: -0.00487 °C per meter, -4.87 °C per 1,000 mintercept: 25.45 °C

So the least-squares line is

$$\text{temperature} = 25.45 - 0.00487 \times \text{height}$$

with the temperature in °C and the height in meters.

Slope and intercept in words

A line’s two numbers mean nothing until you say them in words, with their units:

• The slope is how much the prediction changes for each 1 more of what is across. Here: for every 1,000 m higher, a station is on average about 4.9 °C colder over the year. Multiply to compare: Jumla is about 2,300 m above the Terai stations, so the line expects it to be about 2.3 × 4.9 ≈ 11 °C colder, and it is.
• The intercept is the prediction when what is across is 0. Here: a station at sea level would average about 25.4 °C. Nepal has no land at sea level, so this is the line carried a little beyond the data (Biratnagar, at 72 m, averages 24.9 °C).

Three words in that sentence matter. “On average”: no station sits exactly on the line. “For these stations”: the line describes these 15 stations in these 30 years, not every hillside. And the sentence describes, it doesn’t explain: the reason air is colder higher up comes from physics (air higher up is under less pressure, and air that rises expands and cools), not from the line.

How well does it fit? r²

The temperatures of the 15 stations vary a lot: from 13.1 °C to 24.9 °C. How much of that variation does height account for? Compare two numbers:

• the total spread: add up the squared distances of the temperatures from their mean, 21.01 °C. This is how badly the flattest possible prediction, “every station is 21.01 °C”, would do;
• what is left: the sum of the squared residuals, how badly the line does.

The share of the spread that the line takes away is called r² (said “r squared”), or the coefficient of determination:

$$r^2 = 1 - \frac{\text{left}}{\text{total}}$$

r² = 1 means every dot is on the line; r² = 0 means the line does no better than the flat mean.

import csv heights, temps, names = [], [], []with open("stations.csv") as f:    for row in csv.DictReader(f):        names.append(row["station"])        heights.append(int(row["height"]))        temps.append(float(row["temperature"])) slope, intercept = -0.004866, 25.446       # the least-squares linemy = sum(temps) / len(temps) left = 0        # what the line leaves unexplained: the squared missestotal = 0       # the whole spread: squared distances from the meanfor name, x, y in zip(names, heights, temps):    miss = y - (intercept + slope * x)    left += miss ** 2    total += (y - my) ** 2    if abs(miss) >= 0.4:        print(f"{name:12} {x:5} m  {y:4.1f} °C, the line says {intercept + slope * x:4.1f}, miss {miss:+.1f}") print(f"total {total:.2f}, left {left:.2f}, r² = {1 - left / total:.3f}")

Dipayal        563 m  23.2 °C, the line says 22.7, miss +0.5Okhaldhunga   1731 m  17.5 °C, the line says 17.0, miss +0.5Jumla         2363 m  13.1 °C, the line says 13.9, miss -0.8total 189.57, left 1.79, r² = 0.991

r² = 0.991: height accounts for 99% of the differences in average temperature between these stations. That is unusually high. In most real data, about people, prices or marks, an r² of 0.3 or 0.5 is common, because many things matter at once.

Look at the misses too. Jumla, the highest station, is 0.8 °C colder than the line says. Is that chance, or something the line leaves out? Jumla is a dry valley in the far west, with cold winters; perhaps such valleys are colder than their height alone says. With only one station that high, the data can’t tell, but a pattern in the residuals, such as all the highest stations being below the line, would be a sign that a straight line is the wrong shape.

You may also meet r, the correlation coefficient: a number from −1 to 1 saying how close the dots are to a straight line, and which way it slopes. For one line, r² is r squared, and r has the slope’s sign. Here r = −0.995: a strong correlation, going down.

NumPy, the library pandas is built on, fits the line in one call. np.polyfit(x, y, 1) fits a polynomial of degree 1 (a straight line) by least squares and gives back its slope and intercept; np.corrcoef gives r:

import numpy as npimport pandas as pd st = pd.read_csv("stations.csv")slope, intercept = np.polyfit(st["height"], st["temperature"], 1)r = np.corrcoef(st["height"], st["temperature"])[0, 1]print(f"slope {slope * 1000:.2f} °C per 1,000 m, intercept {intercept:.2f} °C")print(f"r = {r:.3f}, r² = {r * r:.3f}")

slope -4.87 °C per 1,000 m, intercept 25.45 °Cr = -0.995, r² = 0.991

A curve that fits too well

If a line misses Jumla by 0.8 °C, would a curve do better? A polynomial of degree 2 (with x², a parabola) can bend once; one of degree 4 can bend three times; a polynomial of degree 10 has 11 numbers to choose, and can pass exactly through any 11 points. More bends always fit the dots it was fitted to better. The question is whether it predicts better.

To find out, do what Grade 9 and Grade 11 did: keep some data back. Fit each curve to 11 of the stations, the training stations, and test it on the 4 it has never seen: Simara, Dang, Kathmandu and Taplejung, spread over the heights. The typical miss is the square root of the mean squared miss, in °C:

import numpy as npimport pandas as pd st = pd.read_csv("stations.csv")st["km"] = st["height"] / 1000                     # heights in km keep the numbers smallheld_out = ["Simara", "Dang", "Kathmandu", "Taplejung"]test = st[st["station"].isin(held_out)]train = st[~st["station"].isin(held_out)]  def typical_miss(coefs, rows):    """The root of the mean squared miss, in °C."""    misses = np.polyval(coefs, rows["km"]) - rows["temperature"]    return np.sqrt(np.mean(misses ** 2))  print("degree | miss on 11 training stations | miss on 4 test stations")for degree in [1, 2, 4, 8, 10]:    coefs = np.polyfit(train["km"], train["temperature"], degree)    print(f"{degree:6} | {typical_miss(coefs, train):10.2f} °C | {typical_miss(coefs, test):10.2f} °C") coefs = np.polyfit(train["km"], train["temperature"], 10)print(f"Degree 10 says Kathmandu is {np.polyval(coefs, 1.337):.1f} °C (really 19.3 °C)")

degree | miss on 11 training stations | miss on 4 test stations     1 |       0.37 °C |       0.29 °C     2 |       0.21 °C |       0.24 °C     4 |       0.15 °C |       0.31 °C     8 |       0.08 °C |       0.51 °C    10 |       0.00 °C |      17.44 °CDegree 10 says Kathmandu is -14.7 °C (really 19.3 °C)

(np.polyval(coefs, x) works out a polynomial’s value at x. train[~...] keeps the rows where the condition is false: ~ means “not” for a column of True and False.)

Read down the two columns. On the training stations the miss falls with every extra bend, to exactly 0.00 at degree 10, which passes through all 11. On the test stations it does the opposite: from degree 4 on it gets worse, and degree 10 is a disaster. Between the training stations, the curve swings wildly to pass through every one, and says Kathmandu, between Dhankuta and Okhaldhunga, should average 14.7 °C below freezing.

This is overfitting, which you met with neural networks in Grade 11 Grade 11: Overfitting: a model with as many numbers to choose as it has data points can fit every point exactly, chance wobbles and all, and learns nothing that carries over to new data. The fix is the same: judge a model only on data it didn’t see, and prefer the simplest model that does as well.

Did degree 2 really do better than the straight line? On these 4 test stations it missed by 0.24 °C instead of 0.29 °C. That is a difference of 0.05 °C on 4 stations: much smaller than the chance differences you would get by holding back 4 other stations. The data can’t tell the two apart, and there is a physical reason to expect a straight line. A careful analyst keeps the line and says the question is open.

How sure is the slope?

The slope, −4.87 °C per 1,000 m, came from 15 stations. With another 15, it would come out a little different. How different? Resample the stations, as you resampled the households, and fit the line again to each resample:

import randomimport numpy as npimport pandas as pd st = pd.read_csv("stations.csv")stations = list(zip(st["height"] / 1000, st["temperature"]))   # (km, °C) random.seed(3)slopes = []for i in range(1000):    again = random.choices(stations, k=len(stations))   # 15 stations, with replacement    x = [s[0] for s in again]    y = [s[1] for s in again]    slopes.append(np.polyfit(x, y, 1)[0]) slopes.sort()print(f"Slope from all 15: {np.polyfit(st['height'] / 1000, st['temperature'], 1)[0]:.2f} °C per km")print(f"The middle 95% of 1,000 resamples: {slopes[25]:.2f} to {slopes[974]:.2f} °C per km")

Slope from all 15: -4.87 °C per kmThe middle 95% of 1,000 resamples: -5.18 to -4.48 °C per km

So: across these stations, the average temperature falls by about 4.9 °C for every 1,000 m of height (95% interval 4.5 to 5.2 °C). Textbooks of the atmosphere often quote 6.5 °C per 1,000 m, the average fall in the open air far above the ground. Our number is about the air at ground level, at stations mostly in valleys, averaged over all seasons, and studies of Nepal’s stations find it differs by season and region, so it is no surprise that the two differ.

In Grade 7 you met the shop whose ice cream and cold drinks sell together on hot days Grade 7: Finding a trend: two things can go together because a third thing drives both. Two numbers that rise and fall together are correlated. The question that matters for decisions is whether one causes the other: if we change this one, will the other change?

A hidden third factor

Suppose data from many schools shows that students who go to tuition classes get higher SEE marks, on average, than students who don’t. Does tuition raise marks?

Maybe. But think about who goes to tuition. Tuition costs money and time. Families who can pay for it can often also pay for books, a quiet room to study in, a phone with data, a better school; parents who choose tuition may also check homework every evening. Any of these could raise marks with no tuition at all (Figure 5.3).

A factor that affects both things being compared, and so can make them go together, is called a confounder, or a hidden third factor. Correlation can also come from:

• Cause the other way round. People who take medicine are sicker, on average, than people who don’t. The sickness causes the medicine, not the other way.
• Chance. Compare enough pairs of numbers and some will go together by luck (more on this below).
• Who was counted. If only students who passed are in the data, any link you find is about passing students only.

You can split the data by a confounder, as Grade 10 split the hospitals’ patients by how ill they were. But you can only split by the confounders you have thought of and measured. The ones you haven’t can still be there.

How an experiment shows cause

The answer to hidden factors is to stop people choosing. In a randomized controlled experiment (or randomized controlled trial, in medicine), chance decides who gets the treatment, the treatment group, and who doesn’t, the control group. Then every other factor, rich or poor, careful or careless, the ones you thought of and the ones you didn’t, is spread over both groups by chance, and on average the two groups are alike in everything except the treatment. So if the treatment group then does better, by more than chance alone would explain, the treatment caused it.

Two more safeguards are common. The control group often gets a placebo, a pill or treatment that looks the same but does nothing, so that believing you got the treatment doesn’t make the difference. And where possible the experiment is blind: the people in it, and often those measuring the results, don’t know who got what, so their hopes can’t tilt what they report.

Apps run randomized experiments on their users all the time. A bus-ticket app wonders whether a different payment screen would make more people finish buying a ticket. Instead of guessing, or asking, it shows the old screen, A, to some users and the new one, B, to others, chosen at random, and counts what each group does. This is an A/B test: a randomized controlled experiment, with the old version as the control.

Could chance make that difference?

A made-up test. Over one weekend, 400 people who opened the app saw screen A, and 52 of them bought a ticket: 13.0%. Another 400 saw screen B, and 74 bought: 18.5%. B looks better, by 22 tickets. But people are not all alike: even if the screens made no difference at all, two random groups of 400 would not buy exactly the same number of tickets. Could chance alone give 22?

Imagine the screens really make no difference. That idea, that there is no real difference and any you see is chance, is called the null hypothesis. If it were true, each person would have bought a ticket (or not) whichever screen they saw, and the labels “A” and “B” would just be which group chance put them in. So deal the 800 people out again into two groups of 400, at random, and see what difference that makes. Do it many times. This is a shuffle test (statisticians call it a permutation test):

import random # version A: 400 people saw it, 52 bought a ticket; version B: 400 people, 74 boughta = [1] * 52 + [0] * 348b = [1] * 74 + [0] * 326real = sum(b) - sum(a)                        # 22 more tickets with Bprint(f"A: {sum(a) / 400:.1%}, B: {sum(b) / 400:.1%}, B sold {real} more") everyone = a + brandom.seed(4)as_big = 0for i in range(2000):    random.shuffle(everyone)                  # deal the 800 people out again at random    diff = sum(everyone[400:]) - sum(everyone[:400])    if abs(diff) >= real:                     # as big as the real one, either way        as_big += 1print(f"Shuffles with a difference at least as big: {as_big} of 2,000 ({as_big / 2000:.1%})")

A: 13.0%, B: 18.5%, B sold 22 moreShuffles with a difference at least as big: 85 of 2,000 (4.2%)

When the labels mean nothing, a difference of 22 or more, either way, turns up in about 4 shuffles in 100. That share is called the p-value: how often chance alone, with no real difference, would give a result at least as big as the one you got. A small p-value means the result would be surprising if the null hypothesis were true. By a common convention, a p-value under 5% is called statistically significant. Here it is about 4%, so the test is evidence, though not overwhelming evidence, that screen B really sells more.

How many people does a test need?

A good A/B test is designed before it starts: what will be measured (tickets bought, not “anything that looks better”), how many people each version needs, and when it will stop. Stopping the moment the difference looks good is a trap: differences come and go by chance as people arrive, and a test watched every hour and stopped at its best moment will often “find” a difference that isn’t there.

How many people? It depends on how small a difference you want to be able to find. Suppose screen B really is better, with 15% buying instead of 13%. A program can find out: make up 500 tests at each size in which B really is that much better, judge each with a shuffle test, and count how often the test notices. Here is what such a simulation gives:

With 400 people each, a real lead of 2 points is found only about one time in eight: a test that small would usually say “no difference” about a screen that really is better. Even 4,000 each finds it only three times in four. The chance of finding a real difference of a given size is called a test’s power, and many teams aim for about 80%. Small tests can find only big differences: “we found no difference” from a small test is not evidence that there is none.

Most bad statistics aren’t lies. They come from choices that seem harmless: which years to show, how many things to test, which groups to add together. Each choice can make a weak result look strong, and the person making it may not even notice. Here are three of the commonest. (Grade 8 showed how a chart’s design can mislead; these tricks work on the numbers themselves.)

A chosen time period

Here are the average temperatures of each year at Kathmandu airport, from NOAA’s Global Historical Climatology Network, which collects weather records from all over the world (public domain). A year is listed only if all 12 of its months were measured and passed NOAA’s checks, so some years are missing.

year,temperature1951,18.721952,18.541953,18.841954,18.781955,18.621956,18.921958,18.801959,18.401960,18.241961,18.271962,18.061963,18.181964,18.431965,17.841966,18.481967,17.961968,18.131969,18.991970,18.821971,17.391972,17.711973,17.861974,17.831975,17.711976,17.661977,17.431978,17.581979,18.371980,18.381981,17.961983,17.821984,18.371985,18.571986,18.071987,18.361988,18.881989,18.601990,18.491991,18.331992,18.331993,18.521994,18.721995,18.641996,18.841997,17.971998,19.201999,19.712000,19.132001,19.152002,19.162003,19.252004,19.352005,19.502006,20.012007,19.442012,19.332014,18.572015,18.752016,19.632017,19.432020,18.412021,18.532022,18.792023,19.092024,19.27

Now make two claims from the same file:

import numpy as npimport pandas as pd ktm = pd.read_csv("kathmandu_years.csv").set_index("year")["temperature"]print(len(ktm), "years, from", ktm.index.min(), "to", ktm.index.max())  def change(first, last):    print(f"{first} to {last}: {ktm[first]:.2f} °C to {ktm[last]:.2f} °C, a change of {ktm[last] - ktm[first]:+.2f} °C")  change(1971, 2006)      # from the coldest year to the warmestchange(2006, 2020)slope = np.polyfit(ktm.index, ktm.values, 1)[0]print(f"Least-squares line through all {len(ktm)} years: {slope * 10:+.2f} °C a decade")recent = ktm[ktm.index >= 1975]slope = np.polyfit(recent.index, recent.values, 1)[0]print(f"... through the {len(recent)} years from 1975: {slope * 10:+.2f} °C a decade")

65 years, from 1951 to 20241971 to 2006: 17.39 °C to 20.01 °C, a change of +2.62 °C2006 to 2020: 20.01 °C to 18.41 °C, a change of -1.60 °CLeast-squares line through all 65 years: +0.14 °C a decade... through the 42 years from 1975: +0.28 °C a decade

“Kathmandu warmed 2.6 °C in 35 years!” and “Kathmandu cooled 1.6 °C in 14 years!” are both true of this file, and both are misleading. 1971 was the coldest year in it and 2006 the warmest: comparing one year with another measures mostly the luck of those two years. Starting or ending at an unusual point, to get the answer you want, is cherry-picking.

The honest summary uses every year, with a fitted line. Even then the start matters: from 1951 the line rises 0.14 °C a decade, from 1975 twice that, because the 1950s and 60s were warmer than the 1970s. So say which years, and why those.

And ask what else changed. Kathmandu airport’s surroundings grew from fields to a city of millions over these years, and a city holds heat (the urban heat island). Part of this station’s warming may be the city, not the climate. That is why climate scientists compare many stations, in towns and out of them, and correct each one’s record for moves and changes of instrument.

Testing many things until one “works”

A chance result shows up 1 time in 20 at the 5% level, when there is nothing there. So if you test 20 things that do nothing, you expect about one of them to look “significant”. Here are 20 made-up experiments, each comparing the pass rate of 100 students who used a pen of a certain color in the exam with 100 who didn’t. The pens make no difference at all: every student has the same 60% chance of passing.

import random  def shuffle_test(a, b, shuffles=300):    """The share of shuffles giving a difference at least as big as a's and b's."""    real = abs(sum(b) - sum(a))    everyone = a + b    as_big = 0    for i in range(shuffles):        random.shuffle(everyone)        if abs(sum(everyone[len(a):]) - sum(everyone[:len(a)])) >= real:            as_big += 1    return as_big / shuffles  random.seed(2)colors = ["red", "orange", "yellow", "green", "blue", "purple", "pink", "brown", "black", "white",          "gray", "gold", "silver", "maroon", "navy", "teal", "olive", "lime", "cream", "sky blue"]for color in colors:    # 100 students with each color of exam pen, 100 without: the pen changes nothing,    # every student has the same 60% chance of passing either way    with_pen = [1 if random.random() < 0.6 else 0 for s in range(100)]    without = [1 if random.random() < 0.6 else 0 for s in range(100)]    p = shuffle_test(with_pen, without)    if p < 0.05:        print(f"{color} pens: {sum(with_pen)}% passed against {sum(without)}%, chance this big {p:.1%}")print("20 colors tested.")

red pens: 54% passed against 68%, chance this big 4.7%black pens: 66% passed against 50%, chance this big 3.3%20 colors tested.

Two “discoveries”: black pens raise pass rates by 16 points, and red pens lower them. A report that mentions only the black pens, “Students with black pens pass 16 points more often (p = 0.03)”, is reporting chance. Testing many things, or one thing in many ways (many groups, many measures, many cut-off points), and reporting only what crossed the line, is called p-hacking, or data dredging.

The cures: decide before the experiment what you will test and how (some scientists publish this plan in advance, which is called preregistration); report every test you ran, not only the ones that “worked”; when you test many things, demand a much smaller p-value for each (testing 20 things, one simple rule asks for 0.05 ÷ 20 = 0.0025); and test a surprising finding again, on new data, before believing it.

An average that hides groups

Grade 10 met Simpson’s paradox Grade 10: A third variable can turn a result round: a comparison that holds in every group can turn round when the groups are added together, because the groups are different sizes. It is also a favorite way to mislead, often by accident. A hospital reports that its average wait rose after a new booking system came in. But in both the morning and the afternoon shifts, waits fell: the new system brought in many more patients in the busy morning, when every wait is longer, so the average across shifts went up. Whenever two averages are compared, ask what groups are inside them, and whether the mix of groups is the same.

An average can also hide its spread. “The average family in the survey earns Rs. 60,000 a month” can be true when most earn far less and a few earn a great deal: the median, from Grade 10, says more about a typical family.

The methods of this unit work the same on 40 households or 40 million phones. But collecting data about millions of people, all the time, is a different thing from asking 40 households a question. It brings new benefits and new harms, to people who usually never agreed to it and never see it happen.

What it can do

On 25 April 2015, the Gorkha earthquake struck. Within two weeks, Flowminder, a non-profit research group, working with the mobile company Ncell, had used the call records of about 12 million SIM cards, with names and numbers removed, to estimate where people had gone: about 390,000 more people than usual had left the Kathmandu Valley in the days after the earthquake, many to the districts around it and to the Terai. Relief agencies used the estimates to plan where help was needed. No survey could have been done that fast.

Privacy

The data that made that possible says where each phone was, hour by hour, for months. Grade 8 showed that removing names isn’t enough to keep data anonymous Grade 8: Benefits and risks: a few ordinary facts together can fit only one person. At large scale this gets worse, because the more detail kept about each person, the fewer people share any pattern.

In 2006, Netflix, the film-streaming company, published 100 million ratings that about 480,000 of its customers had given to films, with the names replaced by numbers, and offered a prize for better recommendations. Two researchers at the University of Texas, Arvind Narayanan and Vitaly Shmatikov, showed that knowing only a few of a person’s ratings and roughly when they were made, which many people also post publicly on film websites, was enough to pick out that person’s whole list, including films they had rated only in private, which could hint at their politics or religion. A planned second contest was canceled in 2010, after a lawsuit and privacy concerns raised by the US government’s consumer protection agency.

There are better ways than deleting names. Data can be released only as totals over big groups. Or a little random noise can be added to every answer, enough to hide any one person while the totals stay accurate: this is differential privacy, and the United States used it to protect the results of its 2020 census. And the simplest protection of all is data minimization: collect only what is needed for a stated purpose, and delete it when that purpose is done.

Fairness: who is in the data

Big data can feel like it covers everyone. It covers only those it reaches. In 2012 the city of Boston, in the United States, released an app, Street Bump, that used drivers’ phones to detect potholes as they drove over them. Researchers, among them Kate Crawford, pointed out the problem: it would find more potholes in richer neighborhoods, where more people had smartphones and cars, and a city that went only by the app would repair those roads first.

The same holds in Nepal. Phone data doesn’t see people without a phone of their own, often the poorest, the oldest, and many women and children. Online data is mostly from towns. And in AI, the language models of Grade 10 learned mostly from English text on the internet, with very little in Nepali, Maithili or Tamang, so they work best for the people who already have the most Grade 10: How Large Language Models Work. A decision made from data inherits the data’s blind spots: always ask who is missing.

The environment

Every record kept is stored on disks in data centers, copied for safety, and processed again and again; Unit 4 counted their electricity and water Grade 12: Data centers: electricity, water and who pays. Data collected “in case it’s useful one day” and never deleted costs energy every year it is kept. Training large AI models on huge collections of data uses far more. Collecting less, and deleting what isn’t needed, is good for privacy and for the environment at the same time.

To evaluate any large-scale collection of data, ask:

1. What is it for? Is the purpose stated, and does it need this much data, about this many people, for this long?
2. Who agreed? Did people know, and could they say no?
3. Who could be harmed, if the data leaked, were combined with other data, or were used for something else later?
4. Who is missing, and whom would decisions made from it fail?
5. What does it cost in electricity, water and money, and who pays?
6. Who checks? Which law covers it, and who can enforce it?

Everything in this unit comes together in one piece of work: an analysis from a question to a written conclusion, honest about its limits. Here is one, short, done on the weather stations. Yours, in the project below, will be on data of your choice.

1. The question. How much colder is it, on average over the year, for each 1,000 m of height in Nepal, and how sure can we be?

2. The data. The 1991–2020 climate normals of Nepal’s 15 stations in the WMO’s list (above), sent by the Department of Hydrology and Meteorology and published by NOAA’s National Centers for Environmental Information, public domain. Each row is one station: its height (m), its average temperature over the year (°C) and its average rain in a year (mm).

3. Checked and cleaned. 15 stations, no missing values; heights from 72 m (Biratnagar) to 2,363 m (Jumla); temperatures from 13.1 °C to 24.9 °C, all possible. Six stations’ averages are over all 30 years, eight over 28 or 29, and Nepalgunj’s over only 24, as the source’s count of years shows. Nothing needed cleaning, and nothing was changed.

4. A chart of several variables. Height across, temperature up, and the rain as the size of each dot, to see whether wetter or drier places miss the line, with the fitted line drawn:

import numpy as npimport pandas as pdimport matplotlib.pyplot as plt st = pd.read_csv("stations.csv")slope, intercept = np.polyfit(st["height"], st["temperature"], 1) plt.scatter(st["height"], st["temperature"], s=st["rain"] / 15, alpha=0.6,            label="a station (size: rain in a year)")xs = np.array([0, 2500])plt.plot(xs, intercept + slope * xs, color="red",         label=f"least squares: {slope * 1000:.1f} °C per 1,000 m")for name, x, y in zip(st["station"], st["height"], st["temperature"]):    if name in ["Pokhara", "Jumla", "Dhankuta", "Dipayal"]:        plt.annotate(name, (x, y), textcoords="offset points", xytext=(8, 4))plt.xlabel("Height of the station (m)")plt.ylabel("Average temperature, 1991–2020 (°C)")plt.title("Nepal's 15 WMO weather stations")plt.legend()plt.show()

The wettest station, Pokhara, sits right on the line; the driest, Jumla and Dhankuta, sit on either side of it. Rain doesn’t explain the misses.

5. A fitted line, and how well it fits. Temperature = 25.45 − 4.87 × height in km. r² = 0.991. Curves with more bends did no better on stations kept back.

6. The uncertainty. Resampling the stations 1,000 times gives a 95% interval for the slope of 4.5 to 5.2 °C per 1,000 m.

7. The conclusion, with its limits.

Claim. Across Nepal’s main weather stations, the average temperature over the year falls by about 4.9 °C for every 1,000 m of height.

Evidence. The least-squares line through 15 stations’ 1991–2020 averages has a slope of −4.87 °C per 1,000 m, and accounts for 99% of the differences in temperature between them; no station is more than 0.8 °C from it.

How sure. The 95% interval for the slope, by resampling the stations, is 4.5 to 5.2 °C per 1,000 m.

Limits. Only 15 stations, none above 2,363 m, so nothing here says what happens higher up. The stations are mostly in towns and valleys, not on ridges, and were chosen by where weather stations happen to be, not at random. A yearly average hides the seasons: the slope may differ between winter and the monsoon. And the line describes the pattern; the reason, that rising air expands and cools, comes from physics.

A new question it raises: is the slope the same in winter and in the monsoon? The monthly normals in the same source could answer it.