Can a Treatment Win in Every Group but Lose Overall?

Here are the same 700 operations, split by whether the stone was smaller than 2 cm across or larger:

Check: 81 + 192 = 273 and 87 + 263 = 350, so “All” really is the two groups added together. Yet A wins for small stones (93.1% against 86.7%), A wins for large stones (73.0% against 68.8%), and B wins overall.

Each rate inside the table is a conditional relative frequency: the share of successes among one operation’s patients with one size of stone. The “All” column gives each operation’s rate over both sizes, from the table’s marginal totals (A’s 273 of 350).

This happens in real data often enough to have a name: Simpson’s paradox. A comparison that holds in every group turns round when the groups are put together. The third thing that does it, here the size of the stone, is called a lurking variable or a confounding variable: it is linked both to which treatment a patient got and to how the treatment turned out.

It isn’t a contradiction: every number is right. It is a paradox only to our intuition, which expects an overall rate to behave like the group rates it is made of.

Grade 10’s mean adds everything up and divides by how many there are. A success rate is a mean too: give each success 1 and each failure 0, and the mean of the 350 numbers is A’s 78%. So how does A’s 78% come from its two groups?

Of A’s 350 patients, 87 had small stones and 263 large ones. So

$$\frac{273}{350} = \frac{87}{350} \times \frac{81}{87} + \frac{263}{350} \times \frac{192}{263} \approx 0.249 \times 93.1\% + 0.751 \times 73.0\% = 78.0\%.$$

The overall rate is a weighted mean of the group rates: each group’s rate counts in proportion to its share of the patients, its weight. A weighted mean always lies between the smallest and the largest of the rates it mixes, and nearer the one with more weight. A’s 78% lies between 73% and 93%, much nearer 73%, because three in four of A’s patients are in the 73% group. B’s 82.6% lies between 68.8% and 86.7%, near 86.7%, because more than three in four of its patients are there.

So each overall rate is pulled towards a different group. That is the whole trick, and you can work it with your own hands:

With the real patients, B’s dot sits near its small-stone end and A’s near its large-stone end, and B’s overall rate is higher. Move A’s slider down and its dot slides towards 93%: once fewer than about 52% of A’s patients have large stones, A wins overall. And whenever the two sliders are equal, A wins, whatever the share.

Two things together

Look back at what made the reversal. Two things had to happen at once:

• The mix was different. A got mostly large stones, B mostly small ones. If the shares had been equal, the weights would be equal, and A would win.
• The lurking variable mattered more than the treatment. The size of the stone moved the success rate by 18 to 20 points; the choice of operation by only 4 to 6. If the gap between small and large stones had been tiny, no mix could have overturned A’s lead.

Here is the promise in the widget written as a proof. Suppose treatment A beats B in both groups: a₁ > b₁ for small stones and a₂ > b₂ for large ones (each a success rate). Suppose both treatments have the same share w of large stones. Then

$$\text{A overall} - \text{B overall} = (1 - w)(a_1 - b_1) + w\,(a_2 - b_2).$$

Both brackets are positive, and so are 1 − w and w (or one of them is 0), so the difference is positive: A wins overall. With the same mix, a treatment that wins in every group wins overall. The proof works for any number of groups. So a reversal needs different mixes, and that is exactly what the kidney stone patients had.

The computer scientist Judea Pearl, who studies cause and effect, calls this a “sure-thing” theorem: something that raises the chance of success in every group must raise it overall, as long as it doesn’t change how many are in each group.

A picture of the reversal

There is also a picture that shows the reversal with no algebra at all. Draw each group as an arrow: across by the number of patients, up by the number of successes. Then the steepness of an arrow (its slope, up ÷ across) is the group’s success rate. Putting a treatment’s two arrows end to end adds the patients and adds the successes, so the end point is its overall total.

In Figure 29.1, compare A’s and B’s arrows for the same size of stone: A’s is the steeper both times. But A’s long arrow is the shallow one, and B’s long arrow is the steep one. Added end to end, B climbs higher. A reversal is simply two pairs of arrows whose lengths are badly mismatched. The mathematician Jerzy Kocik published this picture in 2001 as a “proof without words”.

Once you know the trick, you can build one. Give the better treatment mostly hard cases.

How small can a reversal be? The smallest has only 9 patients in all (Question d).

Early in the Covid-19 pandemic, two countries reported how many of their confirmed cases had died. China’s Center for Disease Control and Prevention counted 44,672 cases, of which 1,023 died: 2.3%. Italy’s National Institute of Health, on 9 March 2020, counted 8,026 cases, of which 357 died: 4.4%, nearly twice as many. Was the disease deadlier in Italy, or its hospitals worse?

In 2020 three researchers at the Max Planck Institute for Intelligent Systems in Germany, Julius von Kügelgen, Luigi Gresele and Bernhard Schölkopf, split the cases by age. In every age group Italy’s rate was lower than China’s (or the same, at 0, among the young), yet its overall rate was higher. Under 70: Italy 0.9%, China 1.3%. Aged 70 or more: Italy 9.5%, China 9.8%. All ages: Italy 4.4%, China 2.3%.

The weights explain it. Of Italy’s cases, 41% were aged 70 or more; of China’s, 12%. In these 2020 figures Covid-19 was far deadlier for the old, so Italy’s overall rate was a mean weighted heavily towards its most dangerous groups. The researchers suggest why: Italy’s people are older (a median age of 45.4 years against China’s 38.4), and the countries may have tested differently.

Giving everyone the same ages

The widget’s buttons do what health statisticians do every day: to compare two places fairly, they take each place’s rate in each age group and then a weighted mean using one standard population, the same age mix for both. This is called age standardization (or direct standardization). With China’s mix of ages, Italy’s rate comes out at 1.5% against China’s 2.3%; with Italy’s, China’s comes out at 5.6% against Italy’s 4.4%. Either way Italy is lower, as it was in every group.

So far the rule seems simple: always split the data, and believe the groups. It isn’t that simple.

Here are some made-up numbers that Judea Pearl uses, for a new drug given to 80 patients:

The drug loses in both groups and wins overall. Should a doctor give it?

Story 1: the groups are men and women. Being a man or a woman was decided long before the drug was given; the drug can’t change it. In each kind of patient the drug does worse. It looks better overall only because more of those who took it were men, who recover more often anyway. Believe the groups: don’t give the drug.

Story 2: the groups are the patient’s blood pressure after the treatment, high or normal. Now the drug may have caused the split: suppose it lowers blood pressure, which helps recovery. Splitting by blood pressure hides the drug’s main way of working, like judging a fertilizer only among plants of the same height. Believe the total: give the drug.

The numbers are the same in both stories; the right answer is opposite. This is what Simpson himself noticed in his 1951 paper: depending on the story behind the data, the “sensible interpretation” is sometimes in the groups and sometimes in the total. Later work, Pearl’s and others’, gives a rule of thumb that settles most cases:

• Split by something that was there before the treatment and affects the result (the size of the stone, the patient’s age or sex): it is a confounder, and the groups give the fair comparison.
• Don’t split by something the treatment itself changes (blood pressure after the drug): the total is the fair comparison.

Pearl’s full test, the “back-door criterion”, also handles the rarer cases where even something there before the treatment shouldn’t be split by.

No calculation on the table can tell you which story is true. You have to know how the data came about. That is why, for the kidney stones, the split by size is the fairer comparison: the stone’s size was there before the surgeon chose. It still isn’t the last word: the two operations were done in different years, when other things changed too, and both have changed a great deal since 1985, so these numbers are no guide for a patient today.

Before trusting a report that compares two percentages, ask:

• What groups are inside each percentage, and were their mixes the same? Ask for the split by the likely lurking variables: age, how severe, which department, which year.
• Was the split made before or after the treatment? Split by what came before.
• Who chose who got what? If chance chose, the totals are safe from hidden mixes. If people chose, look for what guided them.

• Comparing death rates between countries and years: rates are age-standardized, so that a country or a year with more old people doesn’t look worse only for that. The WHO’s authors wrote in 2001 that direct age standardization “has become the predominant technique in most applications of demography and epidemiology”.
• Medical research: observational studies (where treatments weren’t given at random) report results within groups of age, sex and severity, and adjust for them, because, as Julious and Mullee wrote, the choice of treatment “would have been influenced by patients’ characteristics such as age or severity of condition”.
• Checking for fairness: the Berkeley study is the classic example of asking whether a difference between men and women, or between any two groups, in admissions, hiring or pay, comes from bias or from where each group applied.

• Does Ice Cream Cause Drowning? Two things that rise together because a third thing drives both: the lurking variable again, for numbers that go up and down.
• Can 1,000 People Speak for 3 Crore? How a sample chosen by chance can stand for a whole country, and why chance is the cure for hidden mixes.
• Is a Positive Test Result Bad News? Another case where the proportions of a group change what a percentage means.
• What Does an Average Hide? (Grades 6–8) The Will Rogers phenomenon and other surprises of the mean.
• Grade 10 Math: Mean The mean of grouped data, a weighted mean of its classes’ midpoints.
• Grade 10 Computing: A third variable can turn a result round The same paradox with Python and real data sets.