Skip to content

← Thinking Clearly

Experts and agreement

Chapter 24 · How science decides

In this chapter 5 sections
  1. One study is not the science
  2. A real argument
  3. An argument made to order
  4. Fake journals and fake experts
  5. When to lean, and when to keep asking

The logo of Cochrane, an international network that sums up medical research, is a chart. Two curved letter Cs face each other to make a circle, and inside it are a few horizontal lines, a vertical one, and a small diamond. It looks like a pattern. It is really a summary of a handful of experiments on babies, and a quiet accusation.

The story begins with sheep. In the late 1960s a doctor in Auckland named Liggins was studying what starts labor in ewes. He noticed that lambs born too early after their mothers had been given a steroid hormone lived longer than other early lambs. Babies born too soon often can’t breathe properly, because their lungs aren’t ready, and many of them die of it. So he and his colleague Howie tried the same thing on women, in a fair test: women likely to give birth early got either a steroid injection or a dummy one. Their first report, in 1972, was striking. Among the babies born early, deaths were about a fifth as common after the steroid.

You might expect every hospital to have started at once. It didn’t happen. Other teams ran their own trials over the next fifteen years, and their answers seemed to point every which way. Some found a clear benefit. Others found a smaller one, or none they could be sure of. A survey of British obstetricians in 1980 found that 42% used the treatment frequently, 40% sometimes, and 18% never. Doctors reading the same trials had come to quite different answers.

Then, at the end of the 1980s, Patricia Crowley, Iain Chalmers and Marc Keirse did something that sounds almost too simple. They gathered every trial they could find and looked at them all together. Twelve trials, more than 3,000 women. The answer was no longer in doubt. The steroid cut the odds of a baby dying in its first days by about 40%.

← fewer deathsmore deaths →Auckland, 1972 onBlock, 1977Morrison, 1978Schutte, 1979Taeusch, 1979Papageorgiou, 1979Doran, 1980US Collaborative Group, 1981Schmidt, 1984Morales, 1986Gamsu, 1989All the trials together0.1¼½12odds of a newborn dying (1 = no steroid)
Figure 24.1 The twelve trials Crowley and her colleagues gathered, in order of publication; one had no deaths in either group and can’t be drawn. Each line is a trial’s range of likely answers, and the square marks its best guess, bigger for a bigger trial. Blue lines are wholly on the side of fewer deaths; amber ones cross the dashed line of “no difference”. The diamond is all of them together.

That is the kind of picture in the logo: lines that each, on their own, could be read as “maybe nothing”, and a diamond that couldn’t. The evidence had been sitting in libraries for years. What was missing was someone to add it up. Cochrane’s own explanation of its logo says the treatment “has probably saved thousands of premature babies.” Iain Chalmers, who had the logo drawn in 1992, said the other half. Since the steroids were still not widely used at the time, he wrote, the logo showed “the human costs that can result from failure to prepare systematic, up-to-date reviews.”

One study is not the science

It helps to see why the trials disagreed, because the reasons are the same everywhere.

The first is plain chance. A trial of a hundred babies, in which a few die, has a wide range around its answer, and two such trials can land far apart even when the treatment does exactly the same thing in both. Crowley and her colleagues said this in their paper: because the trials were small, their estimates were “likely to vary widely as a result of the play of chance”, and a doctor reading one of them that found no clear difference might decide that no difference existed. A range that crosses “no effect” means the trial couldn’t tell. It doesn’t mean the treatment failed.

The second is that trials differ. They take place in different hospitals, with different patients, doses and ways of counting. Sometimes a real difference between places hides inside the scatter. Sometimes a trial was simply done less carefully than the others.

The third is that a trial which found nothing much is less likely to reach print at all, so the journals hold a lopsided sample. Crowley’s team knew this, and to find the unpublished trials they wrote letters to 42,000 obstetricians and pediatricians in 18 countries.

Put the three together and you have the reason a single study, however good, is a clue rather than a verdict. The newspaper story about “a new study” almost never says how it fits with the hundred before it. That is the question that matters.

The careful way of answering it is called a systematic review. It works a little like an honest census of the evidence. First you write down the exact question, and how you will search, before you look at any results, so that you can’t quietly choose the studies you like. Then you search everywhere, published and unpublished. You judge how well each study was done, and leave out the ones that weren’t fair tests. Only then, where the studies asked the same question in the same way, do you combine their numbers. That last step, the arithmetic of pooling, is a meta-analysis. A big careful trial counts for more than a small one, much as a survey of a thousand people counts for more than a survey of ten.

The man Cochrane is named after had asked for exactly this. Archie Cochrane was a British doctor and epidemiologist, and in 1979 he wrote: “It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials.” He died in 1988. The first Cochrane Centre opened in Oxford in 1992, and the Cochrane Collaboration began the next year. Its reviews now sum up the trials on questions from vitamin C and colds to steroids before birth.

Nepal has its own place in this kind of adding up. In a trial in Sarlahi reported in 1991, about 28,600 young children were given either a capsule of vitamin A or a dummy capsule every four months. After a year the vitamin A children had died about 30% less often. A trial in Jumla found a fall of 26%. Those were two clues. When eight large trials from Asia and Africa were pooled, the answer was about 23% fewer deaths, and the case looked solid enough to act on. In 1993 Nepal began giving vitamin A capsules twice a year, starting in eight districts, carried by female community health volunteers. By 2002 it reached the whole country, and the program’s own review estimates that it saved more than 45,000 young lives between 2002 and 2017.

A real argument

That isn’t the end of the vitamin A story, and the rest shows what a real scientific dispute looks like.

In 2013 a team led by Shally Awasthi and Richard Peto published DEVTA, a trial in rural Uttar Pradesh, in India, that covered about a million children for five years. That is more children than all the earlier vitamin A trials put together. It found that deaths were only about 4% lower with vitamin A, a difference too small to tell from chance. Its authors wrote that it “contradicts the expectation from other trials that vitamin A supplementation would reduce child mortality by 20–30%”. It had taken them more than six years to publish it.

So which was right, eight trials or one enormous one?

The arguments came at once, in the open, in the letters pages of The Lancet. Alfred Sommer, Keith West, who had led the Sarlahi trial, and Reynaldo Martorell argued that DEVTA wasn’t a real trial of the same kind. The children hadn’t been formally enrolled or followed. The claim that 86% of them got their capsules rested on the logbooks of overworked village workers, though a national survey shortly afterward found only about 6% of young children in Uttar Pradesh had been given vitamin A in the previous six months. Folding such a study into a meta-analysis, they wrote, let one big, weak study drown out the careful ones. Another team, led by Christine Stabell Benn in Denmark, argued nearly the opposite. The eight old trials “were not flawless”, the world had changed since they were run, with new vaccines among other things, and their own unpublished trial in Guinea-Bissau had also found no overall effect. What was needed, they wrote, was new trials, not another sum to reassure the donors.

A Cochrane review of 2022, pooling 19 trials with 1.2 million children, found about 12% fewer deaths, smaller than the early trials suggested, but clearly more than none. Experts still argue over how big the benefit is today, and where. Nepal’s program continues; its own reviewers wrote in 2020 that it would be needed until children’s diets improve.

This is what a real dispute looks like, and it is worth noticing what it is made of. The people arguing are experts in the very question, on every side. They argue in journals, where other experts read them and reply. Their objections are specific: this study didn’t count properly, that one is too old, these places differ. Each objection is the kind that more evidence could settle, and each side says what evidence it wants. The arguments narrow the question over time instead of widening it. And it is about something real and measurable: how much vitamin A helps, and where.

An argument made to order

Set beside it a dispute that was manufactured.

In December 1953, as the evidence that cigarettes cause lung cancer was piling up, the heads of America’s big tobacco companies met at the Plaza Hotel in New York and hired a public relations firm, Hill & Knowlton. On 4 January 1954 a full-page advertisement appeared in more than 400 American newspapers, “A Frank Statement to Cigarette Smokers.” It said that “distinguished authorities” pointed out “that there is no agreement among the authorities regarding what the cause is” and “that there is no proof that cigarette smoking is one of the causes.” It said: “We believe the products we make are not injurious to health.” And it promised a new research committee, run by “a scientist of unimpeachable integrity and national repute.”

The historian Allan Brandt later went through the companies’ own papers. The plan, he found, was “to build and broadcast a major scientific controversy.” The committee’s money went mostly to basic questions in immunology, genetics and cell biology, with “little or no relevance” to whether smoking harmed people. Its scientific advisers had been chosen by the public relations firm. The firm set out to “identify, solicit, support, and amplify the views of skeptics,” and to insist there were two sides in a contentious scientific debate. Newspapers, which like a controversy and want to be fair, often printed the two sides as though they were equal. Fifteen years later, in the memo met earlier in this book, one of the companies put the idea in four words: “Doubt is our product.”

It was nothing like the vitamin A quarrel. The tobacco doubt was aimed at the public, in advertisements and press releases, not at other scientists in journals. Its few doubting experts were sought out, supported and made loud. It never asked for the one study that would settle things, because nothing would have been enough; the demand for more proof was the point. And behind it sat people who would lose a great deal of money if the question were ever closed. The historians Naomi Oreskes and Erik Conway, in Merchants of Doubt (2010), traced the same approach being used later against the science on acid rain, the ozone layer and global warming.

So a few questions tell the two kinds of dispute apart. Where is the argument happening: in the journals, among people who study this, or in ads, videos and talk shows? Are the doubters experts in this question, or in something else? Does each side say what evidence would change its mind? Who pays, and what do they stand to lose? And is the argument about the size of something, or about whether anything at all is known?

It is the question to ask about climate. When Mark Lynas and his colleagues took a random sample of 3,000 of the 88,125 climate papers published since 2012 and read what each said about the cause of the warming, they found four that doubted humans were causing it. More than 99% agreed.

Fake journals and fake experts

Behind most of this sits an assumption that a study in a journal has been checked by other experts before it was printed. That checking, called peer review, is imperfect, but it is something. Unless, of course, the journal is a fake.

Some journals exist mainly to collect fees from the authors who publish in them, and do little or no checking. They are called predatory journals. In 2013 John Bohannon, writing for the journal Science, made up a paper about a molecule from a lichen that stopped cancer cells growing, made it obviously flawed on purpose, and sent versions of it, under invented names from invented institutes, to 304 journals that charge authors a fee. By the time he wrote it up, 157 had accepted it and 98 had rejected it. Of the journals that showed any sign of reviewing it at all, only 36 pointed out its problems, and 16 of those accepted it anyway.

A few years later, researchers in Poland invented a scientist called Anna O. Szust. Oszust is Polish for “a fraud”. They gave her a CV with no published papers, books and chapters that didn’t exist, and publishers that didn’t exist either, and applied on her behalf to become an editor at 360 journals. Of the 120 drawn from a list of well-established journals, none took her. Of the 120 suspected predatory journals, 40 did, often within days or even hours. Four journals made her editor-in-chief at once. At least a dozen wanted money, or a share of the profits, along the way. One asked for a fee of $750, then cut it to “ONLY $650”.

These fakes matter because they borrow the look of the real thing. A claim with a journal name and a reference number after it looks like science. A person called “Dr.” or “researcher”, with a page of titles, looks like an expert. Neither look is hard to buy. What can’t be bought so easily is a record: a journal that the field itself reads and cites, a person whose work on this very question other experts have checked, and the agreement of many such people working separately.

When to lean, and when to keep asking

None of this means checking everything yourself. You can’t, and neither can anyone else. A cardiologist takes the geologists’ word on earthquakes, and the geologists take the cardiologist’s word on hearts. The sensible thing, for anyone who isn’t an expert in a question, is usually to lean on the people who are, in proportion to how far they agree.

Lean hard when the experts in that very field broadly agree, when they reached it by different routes, when the field has a good record on questions of this kind, and when the evidence is the sort anyone could, in principle, check. Then your own doubt, however clever it feels, is very unlikely to be right where theirs has failed. That is why it is reasonable to believe that steroids help early babies, that vitamin A saved children in Nepal, and that smoking causes cancer, without having read a single trial.

Keep asking when the agreement is thin or new, when a single study is all there is, when the expert is talking outside their own field, when money or a cause stands behind the voice, or when the journal or the title can’t be traced to a real record. Keep asking too when the question is really about what to do, rather than what is true: experts can tell you what a choice is likely to cause, but not always what is worth choosing.

And agreement can be wrong. For most of the twentieth century, doctors were sure that stomach ulcers, what many people in Nepal call “gastric”, came from stress, spicy food and too much acid. In the early 1980s two Australians, the pathologist Robin Warren and the young doctor Barry Marshall, reported a spiral bacterium in the stomachs of ulcer patients. Most of their colleagues didn’t believe it. To prove it, Marshall swallowed a broth of the bacteria and made himself ill. What settled it in the end wasn’t the stunt, though. It was trials showing that when antibiotics killed the bacteria, ulcers healed and stayed away. In 1994 a panel at America’s National Institutes of Health concluded that ulcer patients with the infection needed antibiotics, and in 2005 Warren and Marshall shared the Nobel Prize.

How that happened matters. The consensus didn’t give way to a loud outsider or a viral video. It gave way to evidence, published, tested, repeated and weighed, by the same machinery that had built the consensus in the first place. Someone who doubted the doctors on ulcers in 1980 for no particular reason wasn’t right in any useful sense; they were lucky. The person who was right was the one who changed their mind when the trials came in.

That is what agreement among experts is worth. It isn’t certainty, and it isn’t a vote. It is the best bet anyone has, made by the people who have looked hardest, and it changes when better evidence comes. Think of the diamond in the Cochrane logo. None of the lines above it is the truth on its own. Together, honestly added up, they are the nearest thing to it, and they can still be added to. Even the logo’s own story has been: in 2015 a large trial in poorer countries, of a push to give the steroid far more widely, saw more newborn deaths rather than fewer, and Chalmers asked in print whether the logo now needed a health warning.

Found a mistake? Please tell us.