In the villages of Sarlahi, between late 2002 and early 2005, project workers went from house to house with small plastic bottles. At each house with a new baby the worker washed their hands, wet a cotton ball from the bottle and dabbed the stump of the umbilical cord. Some bottles held an antiseptic called chlorhexidine. Others held mild soap and water. Soap has a scent, so the team, filling the bottles in Kathmandu, added the soap’s perfume to the antiseptic. Now the two smelled the same, and they looked the same. The worker holding a bottle didn’t know what was in it, and neither did the mother or the people who would later count the results.
That’s a lot of trouble over a dab of liquid. These people wanted to save babies. Why take such care that nobody could tell the bottles apart?
Because this is how you find out whether something really works. A story can’t tell you. Most people get better anyway, and two things happening together don’t show that one caused the other, so “my grandmother took it and got better” proves nothing, even if it’s true. What can tell you is a fair test, and a fair test comes down to four ideas.
How to find out if something works
First, compare two groups. One group gets the remedy and the other doesn’t. The second group is called the control group, and its job is to show what would have happened anyway. Without it, all you know is that some people got better.
Second, let chance decide who gets what. Toss a coin for each person. If people choose for themselves, or you choose for them, the two groups will differ in ways that have nothing to do with the remedy. A coin makes them alike.
Third, keep people from knowing who got which. If a doctor knows which child had the real thing, she may look harder at one group than the other, and then the groups aren’t being treated alike anymore. Hiding it is called blinding.
Fourth, count the result honestly. Count everyone the coin chose, whether or not they did as planned, and say how sure you are.
So: two groups, a coin, a secret and an honest count. A test built this way is called a randomized trial, and it’s the best answer anyone has found to the question every claim of a cure has to face: compared with what?
The coin is the part that’s hardest to believe. Suppose someone wants to test हिमाली बुटी चिया, the miracle tea from the forwarded video. (Both are made up for this book.) The natural way is to offer it to villages and let them decide. The tea then reaches villages near a road, where the seller’s truck can go. Those villages also have a health post closer by, and more money. They were healthier before anyone brewed anything. Compare tea drinkers with the rest, and the tea gets the credit for the road.
With a tea that does nothing, the villages that chose it come out about 6 people in 100 healthier, test after test. That gap isn’t luck. It stays however often you run the test, because it comes from the way the groups were made. Let the coin choose, and the road villages fall about evenly on both sides. On average the gap disappears.
Leaving it to chance sounds like giving up, but it’s the opposite. You could share out the road villages by hand, since you know about the road. That does nothing for clean water or a good midwife, or whatever else matters that nobody has thought of. A coin doesn’t need to know what matters. It ignores everything alike, so everything gets shared out alike, the things you measured and the things you never imagined. (Real trials use a computer more often than a coin, but the idea is the same.)
It isn’t perfect. Chance can deal one group a few more road villages. That’s why a trial’s answer comes with a range around it, and why bigger trials are better: the more villages, the more evenly the coin shares things out.
Why nobody may know
A coin makes two groups alike at the start. It can’t keep them alike afterward. Imagine parents who know their child got nothing. At the first fever they may rush to the clinic, and a doctor who knows may order a blood test sooner. Now the groups differ in how hard anyone looked.
Lalitpur’s typhoid trial shows how far people will go to prevent that. Typhoid is common in the Kathmandu Valley, and a new kind of vaccine existed, but nobody yet knew how it did in a place where typhoid is at home. From November 2017 to April 2018, a team from Patan Hospital and Oxford enrolled 20,019 children. A program on a tablet decided each child’s vaccine. Half got the typhoid vaccine. The rest didn’t get a dummy. They got a real vaccine against one kind of meningitis, so that every child had a real injection and nobody could tell by the experience. Only the staff who gave the injection knew which it was, and they took no part in following the children afterward.
Nobody’s opinion came into the count either: a case was typhoid only if a blood test grew the bacterium. In about a year the result was 7 cases against 38, so the vaccine cut typhoid by about 80%, with a range well clear of zero. So the whole answer rests on a few dozen children out of twenty thousand. That’s why a trial needs so many: most children would never have caught typhoid either way.
The control group did a second job, the one people forget. In the week after the injection, 5.0% of the typhoid-vaccine children had a fever. One child in twenty feverish after a new vaccine sounds alarming. In the control group, given the meningitis vaccine, it was 5.4%. So the new vaccine brought no more fevers than the other. Only a second group can show that.
A coin can only be tossed fairly over something nobody knows the answer to, and once the answer is in, the people who helped find it are owed the benefit. In April 2022 Nepal began giving a typhoid vaccine of this kind to children.
Why everyone gets counted
In Makwanpur, at the start of this century, the question wasn’t about a medicine. Could women meeting to work on their own problems save newborn babies?
A local woman, trained for the job, ran monthly meetings of women’s groups. They named the problems around birth in their own place, then decided what to do: clean delivery kits, funds for emergencies, stretchers to carry the sick. You can’t easily offer that to one woman and keep it from her neighbor, so the coin was tossed for whole areas of about 7,000 people each. In each of twelve pairs of areas, chance picked the one that got the groups. Both sides got the same other help, so the groups were the only planned difference.
From 2001 to 2003, the areas with groups saw 26 babies die in their first month for every 1,000 born alive. The others saw 37. Now the odd part. Only 8% of the women in the study joined a group. Most of them weren’t pregnant at the time, and the groups did draw 37% of newly pregnant women, but most mothers never came.
The team counted every birth anyway, joiner or not. Why not compare just the women who joined with the areas that had no groups? Because the coin chose the areas. It didn’t choose who walked to the meeting. Women who join may differ from women who don’t, so comparing them takes you right back to the villages near the road. The rule has a dull name, intention to treat, and a simple meaning: you count people by what the coin chose, not by what they went on to do. It gives up some sharpness and keeps the fairness.
A woman knows whether her village has a group, so there was no keeping that secret, and the result was clearer about whether the groups worked than about why. Real trials often have to give up something. The good ones say what.
What the bottles found
Back to Sarlahi, where all four ideas were at work. A computer dealt 413 small communities into three groups: chlorhexidine, soap and water, and dry care, which meant no wash at all. Dry care was what the World Health Organization (WHO) then advised, so it was the fair standard to beat. In all, 15,123 babies took part.
Here is why the perfume mattered. Anyone could see which babies were getting dry care, since nobody touched the cord. If chlorhexidine had only been compared with that, any good it did might have come from the gentle cleaning itself, or from how workers behaved with a bottle in hand. Soap and water got the same dabbing, and it showed no sign of helping with infection or deaths. So whatever chlorhexidine did, it wasn’t the washing.
And it did something. Severe cord infection appeared in 13 babies, against 52 with dry care, about 75% fewer. In the chlorhexidine group 14.6 babies in every 1,000 died, and with dry care 19.3, which is 24% fewer.
That’s where the honest count comes in, because 24% was only the likeliest answer. Around it lay the range of answers the data couldn’t reasonably rule out, which scientists call a confidence interval. It ran from 45% fewer deaths to 4% more. So the trial didn’t prove that chlorhexidine saves lives, and it didn’t show that it does nothing. It said: it probably helps, perhaps a lot, and this one trial can’t rule out no effect at all. A range that reaches past “no effect” doesn’t mean there is no effect. It means the question isn’t settled yet.
Chlorhexidine costs little and is simple to use, so hopes ran high. Trials in Pakistan and Bangladesh followed, and together the three South Asian trials gave 23% fewer deaths, with a range clear of zero. The WHO changed its advice for babies born at home in places where many newborns die. Nepal hadn’t waited. In 2011 it became the first country to bring chlorhexidine in nationally, and by 2016 about half of its newborns were getting it.
Then in 2016 the answers came from Africa. Two trials, in Zambia and on Pemba Island in Tanzania, each bigger than Sarlahi’s, found no fall in deaths that could be told from chance.
So was Sarlahi wrong? Not exactly. On Pemba the antiseptic did its job on the cord just as it had in Nepal: severe infection fell from 33 babies to 2. But fewer infected cords didn’t become clearly fewer deaths. Both trials saw far fewer deaths than they had planned for, which left them too small to be sure of a modest effect. And the places were different. In the South Asian trials most babies were born at home, and in Nepal and Pakistan many had something else put on the cord, such as mustard oil. In Zambia most were born in a health facility, and few had anything else applied. The experts didn’t agree on which of these mattered most.
In 2022 the WHO looked at all five trials together, about 137,000 newborns. Overall, chlorhexidine made little or no difference to deaths. But where usual care put something harmful on the cord, deaths were 37% lower. So the WHO, rating the evidence low in certainty, now advises clean, dry care, with chlorhexidine only where harmful things such as animal dung are commonly used. In Nepal’s 2016 survey more than one newborn in five had something like oil or ash on the cord, so what goes on a cord is still worth asking about here.
Nobody cheated, and nobody was foolish. All the trials were fair. A fair test tells you what happened to these people, in this place, at this time, and how sure you can be of that. It doesn’t promise the same answer elsewhere. Each new place asks the question again.
Asking it of the video
Take any claim of a cure, such as the video’s “95% of users got better”. Compared with what? Who chose to drink the tea? Who knew what they were drinking, and who decided what “better” meant? Who got counted, and how wide is the range? A real remedy can answer all of these. Chlorhexidine for cord infection did, and so did a vaccine in Lalitpur and women working on their own problems in Makwanpur.
A fair test is slow and costly, and it often hands back a range rather than a yes. Sometimes it can’t be done: nobody can toss a coin to decide who smokes for thirty years. But think of that bottle in Sarlahi, with its borrowed perfume. Anyone who carries a remedy to a door hopes it will work. That’s why the bottle shouldn’t tell them what’s in it.
Found a mistake? Please tell us.
