In the 1990s, in a psychology laboratory at Case Western Reserve University in Ohio, someone baked chocolate chip cookies in a small oven right there in the room. The students who came in, one at a time, had been asked to skip a meal first, so they walked hungry into the smell of fresh baking. On the table were two foods: a stack of the cookies with some chocolate sweets, and a bowl of red and white radishes. Some students were asked to eat the chocolate. Others were asked to eat radishes, and only radishes, while the experimenter left the room and watched through a one-way mirror.
Then everyone got puzzles: trace a figure without lifting the pencil or going over a line twice. The puzzles couldn’t be solved, though nobody told the students that, and the question was how long each would keep trying. Those who had eaten chocolate gave up after about 19 minutes on average. A third group, who had skipped the food part, lasted about 21. The radish eaters quit after about 8.
The four psychologists who ran it, led by Roy Baumeister, published it in 1998 with an idea to explain it. Self-control, they suggested, draws on a store that runs down. Spend some of it resisting cookies and there’s less left for a frustrating puzzle. They called it ego depletion. It was a lovely idea, simple and useful, and other laboratories kept finding it. By 2014 there were more than a hundred studies, and it was being offered as part of the reason people overspend and overeat.
Then, in 2016, 23 laboratories ran one agreed version of the test together, on 2,141 people. The effect came out so close to zero that it couldn’t be told apart from nothing at all. In 2021 another team tried with 36 laboratories and 3,531 people, and the test it had planned in advance found nothing it could tell apart from zero either. Some researchers still think a small one is there. Nobody found the one that a hundred studies seemed to show.
How can a hundred studies by careful people show something that, tested properly, hardly seems to be there? Lying is one answer, and the rarest. The usual answer is duller, and it’s built into how science gets done.
Any study has some luck in it. Put 20 people in one group and 20 in another, give them the same puzzle, and the two groups will never quit at exactly the same moment, even if nothing at all is different about them. So scientists ask a particular question of their results: if nothing were really going on, how often would chance alone produce a gap at least as big as this one? If the answer is less than one time in twenty, the result is called statistically significant. By long habit, that has been the line a result must cross to count as a finding.
One in twenty sounds safe. But it means that if twenty teams test an idea that’s false, about one of them will get a “significant” result by luck alone. On its own that’s no disaster, as long as all twenty report what they found. The trouble starts with two very human things. Results that show something are far more welcome than results that show nothing, to journals and to the scientists themselves, so the lucky team publishes and the other nineteen may not. And every team has many ways to look at its data, and can keep looking until something shows. Put the two together and a false finding can fill a shelf. Once in a while there’s a third thing: someone makes the data up.
The drawer
In the United States, before a new medicine can be sold, the company that makes it sends its trials to the drug regulator, the Food and Drug Administration (FDA), good results and bad. In 2008 a team of researchers took the FDA’s reviews of 74 such trials, of twelve antidepressants and 12,564 patients, and went looking for the same trials in the medical journals, where doctors read about them.
In the FDA’s files, 38 trials had positive results and 36 had results that were negative or doubtful. Of the 38 positive ones, 37 were published. Of the 36 others, 22 were never published at all, and 11 were published in a way that, in the team’s judgment, made them look positive. Only 3 appeared as what they were. So a doctor reading the journals would have seen 94% of the trials come out well, when in the full files it was 51%. Taken together, the journals made the drugs look about a third more effective than the files did.
The team couldn’t tell who had left the trials out: the authors and companies who never sent them, or the journals that wouldn’t print them, or both. And it isn’t only medicine, or only companies. A study of 221 social science experiments, all approved in advance by the same American program so that every one of them could be traced, found that strong results were 40 percentage points more likely to be published than results that showed nothing, and 60 points more likely even to be written up. Most of the empty results never left the researchers’ desks. Scientists call this the file drawer.
Who pays matters too. Across 75 studies of the question, which the reviewers rated as moderately strong evidence, research on drugs and medical devices paid for by the company that makes them came out favorable more often than research paid for by others, about a quarter more often in its results and a third more often in its conclusions. These company studies weren’t sloppier by the usual checks. Their blinding was, if anything, better. The bias was there anyway, and the usual checklists couldn’t find where it came from. None of this means medicines don’t work, or that a study is wrong because a company paid for it. Most new medicines are tested in trials their makers pay for, and many of those trials are careful and honest. The lean shows across many studies, not as a verdict on any one. What it means is that what reaches print is a selection, and the selection leans one way.
Fishing
In 2011 three psychologists, Joseph Simmons, Leif Nelson and Uri Simonsohn, published a study showing that listening to the Beatles’ song “When I’m Sixty-Four” makes you younger. Not feel younger. Be younger. Twenty students at the University of Pennsylvania listened either to that song or to “Kalimba”, an instrumental tune that came free with Windows 7, and then wrote down their date of birth. Allowing for their fathers’ ages, the Beatles group came out about a year and a half younger.
It was a joke with a point. Every step was real and honestly reported, and that was what the authors wanted to show: you can get an impossible result through ordinary choices that each look reasonable. Later in the same paper they showed what they had left out. They had asked the students about ten other things, from how old they felt to their politics, and reported only the father’s age, which turned out to be what made the result significant. Without it there was no significant difference. They had checked the results after every ten or so students instead of deciding in advance when to stop. And there had been a third song, “Hot Potato” by the Wiggles, which they quietly dropped.
Each of those choices is common. With a computer they worked out how much each one raises the chance of a false finding when nothing at all is going on, against the 5% that “one time in twenty” promises. Measuring two things and reporting whichever works nearly doubled it. Using four such choices together took it to 61%. A researcher using them was more likely than not to find something that wasn’t there.
This is sometimes called p-hacking, after the number that measures the chance, and it needn’t feel like cheating from the inside. The statisticians Andrew Gelman and Eric Loken pointed out that a researcher may run only one analysis and still be fishing, because the analysis is chosen after seeing the data: which people to leave out, which measure to use, which groups to compare. Had the data come out differently, a different choice would have seemed just as sensible. They called it the garden of forking paths, after a story by Jorge Luis Borges, and wrote that “whatever route you take seems predetermined”. The simplest version needs no garden at all. Measure 20 unrelated things in a study where nothing is going on, and the chance that at least one of them crosses the line by luck is about 64%.
That is how a finding like ego depletion can grow a hundred studies without anyone lying. A lab that tests it and gets nothing may decide its version of the test wasn’t quite right, and put the result in the drawer. A lab that gets something publishes. The reviews that later went back over the ego-depletion studies found signs of this kind of bias in the published record.
When the data is made up
Most scientists never fake anything. When 18 surveys of scientists were combined in 2009, about 2% of scientists admitted to having made up or falsified data at least once, and about 14% said they knew a colleague who had. Those are only the ones willing to say so, but even taken as they are, faking is the rare way for a finding to be wrong. It’s also the most damaging, because it can’t be fixed by more care.
Diederik Stapel was a senior social psychologist at Tilburg University in the Netherlands, in a position the investigators later called virtually unassailable. He worked out experiments with his students and colleagues properly, in detail, down to the questionnaires and the sweets to hand out as rewards. Then, by his own account to the committees who later investigated him, the boxes of questionnaires he loaded into his car never reached the schools where the experiments were supposed to happen. They were thrown away, and he sat at his computer with a blank questionnaire and typed in the data himself. The results he handed back almost always confirmed what the team had hoped for. The committees established fraud in 55 of his publications, and 58 of his papers have now been withdrawn.
He wasn’t caught by the journals. The committees found the data had been “too good to be true” in plain sight, with impossible numbers that reviewers never noticed. He was caught by three young researchers in his own department, who watched for months, gathered evidence and reported him in August 2011, though they had much to lose. Others had raised doubts earlier and nothing had happened. The committees wrote that the three “showed more courage, vigilance and inquisitiveness than incumbent full professors”.
A faked paper can do real harm before it’s withdrawn. In 1998 The Lancet published a paper by the British doctor Andrew Wakefield, based on 12 children, which linked the measles, mumps and rubella (MMR) vaccine with autism. In many countries fewer children were given the vaccine. The Lancet retracted the paper, which means formally withdrawing it, in 2010, twelve years later. The same year Britain’s medical council struck Wakefield off its register, and in 2011 the journalist Brian Deer showed that not one of the 12 children’s cases in the paper was free of misrepresentation or undisclosed alteration. Retraction is how science marks a paper as not to be trusted, and it’s getting more common: in 2023 the number of retractions passed 10,000 for the first time.
Trying it again
The real test of any finding is to do it again. In 2011 a group of psychologists led by Brian Nosek set out to repeat 100 studies, all published in 2008 in three of psychology’s leading journals, using the original materials where they could and designs big enough to catch the original effects. Their results came out in 2015, and the numbers need saying exactly, because they’re often rounded into something they aren’t.
Of the 100 original studies, 97% had reported significant results. Of the repeats, 36% did. In 39% the team judged that the repeat had found what the original found. And on average the effects in the repeats were half the size of the originals. That’s the source of the line you may hear, that “only a third of psychology replicates”. It’s roughly what the project found, by the measures it used.
Even that was disputed. Four researchers, among them Daniel Gilbert of Harvard, wrote that the project contained three statistical errors and that its data were consistent with the opposite conclusion: that psychology’s findings repeat quite well. Some repeats, they said, were very different from the originals. In one original study, white students at Stanford watched a video of Stanford students discussing their university’s admissions policy. The repeat was run in Amsterdam, with Dutch students watching the Stanford students on video, speaking in English about a university more than 5,000 miles away. They argued that the number of failures was about what chance alone would produce, and that repeats which stayed close to the original failed far less often than those that didn’t. The project’s team answered that the critics had misread the statistics, and that from these data both hopeful and gloomy conclusions were possible, and neither was yet justified.
So other projects asked the question differently. One, called Many Labs, ran 13 findings in 36 places at once, and 10 held up clearly. A second, Many Labs 2, took 28 findings and ran each in around 60 of its 125 samples, 15,305 people in 36 countries, with the plans checked before any data came in. Fifteen of the 28, 54%, held up. The typical effect was about a quarter of the size first reported. And whether a finding held depended mostly on the finding, not on where it was tested. A finding that held in one country tended to hold in the others, and one that failed rarely came back to life somewhere else.
None of this means two-thirds of psychology is false. A failed repeat can be the repeat’s own bad luck, and a finding at half its first size can still be true. What it means is that one published study, especially a surprising one, is weaker evidence than it looks.
The fix came from inside
Most of the people in this chapter who found the problems were scientists. Stapel was caught by his own junior colleagues. The Beatles paper was written by psychologists, about psychology. The mass repeats were run by psychologists, on their own field’s published results. In 2012 Daniel Kahneman, a psychologist with a Nobel Prize, wrote to colleagues whose findings he had relied on in his own recent book. He called himself “a general believer” in their work and told them, “I see a train wreck looming.” He asked them to repeat each other’s studies, publicly and by an agreed plan.
What changed is mostly the timing of decisions. In preregistration, scientists write down before they collect any data exactly what they will measure and how they will analyze it, and put that plan where others can see it. Then the forking paths are chosen before anyone knows which way the data runs. Medicine got there first. In 2004 the major medical journals agreed that, from the middle of 2005, they would publish a trial only if it had been registered before its first patient joined. Large American heart trials paid for by the government had found a benefit in 17 of 30 cases (57%) before 2000, and in 2 of 25 (8%) after, and registration was strongly linked with the change. The authors think registration may be part of the reason: with the measure fixed in advance, there was less room to find something. Closer to home, the typhoid vaccine trial in Lalitpur, with 20,019 children, is listed in a public trial registry, and its first report says the analysis it gives was set out in advance.
A registered report goes one step further. The journal judges the plan, before the results exist, and promises to publish whatever comes out, so the drawer has nowhere to be. The journal Cortex started offering it in 2013, and today more than 300 journals do. When researchers compared the two kinds of paper in psychology, 96% of ordinary papers supported the first idea they tested. In registered reports it was 44%. That gap is about the size of the problem.
The last change is open data, putting the numbers themselves online. A project that tried to repeat 193 experiments from important cancer-biology papers found that the data needed to plan a fair repeat was publicly available for just 4 of them. Data in the open is also data someone can check: once the investigators’ statisticians had Stapel’s data in front of them, the signs of invention were there to find. The Reproducibility Project kept its whole project open online, and its critics and defenders argued from the same public results.
Nobody can say the radish eaters didn’t quit sooner on that day in Ohio. They did. What we have learned is how little one room on one day can tell us, and the people who taught us that were scientists who checked their own work and published what they found, including the parts that hurt.
Found a mistake? Please tell us.
