Skip to content

← Psychology

Who was in the study

Chapter 3 · How we know

In this chapter 5 sections
  1. A thin slice of humanity
  2. Where WEIRD people stand out
  3. Did it happen again?
  4. One test, many countries
  5. What it means in Nepal

Open almost any psychology paper and, a page or two in, under a heading such as Participants, there is a short paragraph saying who was studied. Very often the answer is a few dozen undergraduates at an American university. In one of the field’s leading journals, two-thirds of the American samples were made up only of students taking psychology courses. So when a book tells you that “people” remember this or decide that, the people were often American university students who had come in for an hour in a laboratory.

That gives two questions to ask of any finding in this book, and this chapter is about both. The first is who was in the study, and whether what was true of them is true of you. The second is whether the finding came out the same when someone else did it again. The short answers are that most of psychology was worked out on a thin and unusual slice of humanity, that some of what it found holds for people everywhere while some doesn’t, and that a good part of it, when tested again, came out smaller or not at all. For a great deal of it, nobody has ever looked in Nepal.

A thin slice of humanity

The American psychologist Jeffrey Arnett once counted. He went through five years of articles, those published from 2003 to 2007, in six of the American Psychological Association’s leading journals and noted where the people in each study came from. About seven in ten were Americans. Nearly all the rest were from Europe, Canada, Australia and Israel. Americans are fewer than one person in twenty on Earth, and the Western countries together about one in eight.

United Statesother Western countriesthe rest of the worldThe world’s people88%5%7%People in the studies, 2003–0768%28%4%
Figure 3.1 Where the world’s people live, and where the people in the studies of six leading American psychology journals came from, 2003 to 2007: Arnett’s count, as Henrich, Heine and Norenzayan report it. Each bar adds up to 100%.

Two years later three psychologists at the University of British Columbia in Canada, Joseph Henrich, Steven Heine and Ara Norenzayan, put a name to the problem. The people psychology studies, they wrote, are WEIRD: from societies that are Western, Educated, Industrialized, Rich and Democratic. They worked out from Arnett’s numbers that an American undergraduate was more than 4,000 times more likely to be in a psychology study than a person chosen at random from outside the West.

A decade later, his own team counted the same six journals again, for the five years to 2018. American samples had fallen to a little over 60%. Most of the change came from more studies in other English-speaking countries and in Western Europe. People from the rest of the world, where most humans live, still made up only 4 to 5% of the samples.

None of this would matter if people everywhere thought and felt alike, so that American students could stand in for everyone. That is what many psychologists had quietly assumed. The question is whether it’s true.

Where WEIRD people stand out

Henrich and his colleagues went looking for every study they could find that had compared people from very different societies on the same task. Again and again, they reported, WEIRD people turned up at one end of the range, not in the middle.

Their clearest case is a game played for real money. Two strangers who never see each other are given a sum of money. One of them, the proposer, offers some share of it to the other, who can accept or refuse. If the second player accepts, the money is split as offered. If she refuses, neither gets anything. A purely selfish person should accept any offer, since a little is better than nothing, and so a selfish proposer should offer as little as possible. Students in America, Europe and Asia don’t play that way. They usually offer between 40% and half, and an offer below about 30% is often refused, even though refusing costs the refuser money.

For years this was taken as a window on human nature: a built-in sense of fairness and a readiness to punish unfairness at a cost. Then teams of anthropologists and economists took the game to 23 small societies of foragers, herders and farmers in Africa, the Amazon, Siberia, New Guinea and Oceania. Some played like American students. Many didn’t. In a version of the game where the second player can’t refuse, American adults gave away nearly twice as much as the Hadza foragers of Tanzania or the Tsimane of the Bolivian Amazon. In the version with refusal, Americans and the Sursurunga of Papua New Guinea were the readiest to punish a low offer. The societies whose people traded more in markets, and those where more people belonged to one of the world religions, tended to offer more. The sense of fairness was real, but how much of a stranger’s money is fair turned out to depend on where you grew up.

The same pattern turned up elsewhere. Westerners, and Americans most of all, tend to see themselves as better than average, and psychology long treated this as a basic part of being human. In many studies in East Asia the tendency was absent, or ran the other way, with people rating themselves more modestly than others. Even something as homely as where a baby sleeps can show how unusual the standard subjects are. In an old survey of a hundred societies, American parents were the only ones who put a baby to sleep in a room of its own.

Henrich’s team even argued that eyesight could differ, with Westerners raised among straight walls and square corners more fooled by a famous illusion of two lines with arrowheads. That example has since come under heavy attack, and this book comes back to it when it reaches the senses.

The three authors were careful to say that they weren’t claiming people share little. Their list of what does seem to hold across very different societies is long: seeing colors, a rough sense of quantity without counting, and the dawning understanding that other people can believe things that aren’t true, which children everywhere they looked came to, though anywhere from age four to nine or later, with Western children at the early end. Their point was narrower and harder to dodge. You can’t tell which findings are universal by testing one kind of person. You have to go and look.

Did it happen again?

The second question is older and simpler. A finding becomes something to lean on only when someone else does the study again and gets the same answer, the checking step that Oskar Pfungst took when he played the horse himself with 25 people in his laboratory. For a long time psychology mostly skipped that step. Repeating someone else’s study was dull work, and journals preferred new results.

So a large group of psychologists, the Open Science Collaboration, set out to repeat 100 studies published in a single year in three of the field’s leading journals. They used the original materials wherever they could, and enough people to catch the original effects. Of the 100 original studies, 97 had reported a clear result. In the repeats, only about a third did, and the effects found were on average half the size. Findings about memory, attention and thinking held up better than findings about social life: half of the first kind came out clearly again, a quarter of the second. And the more surprising a finding had seemed, the less often it came back.

Why so much fails, how a hundred careful studies can show something that isn’t there, and the argument over how gloomy to be about these numbers, are told in Thinking Clearly, in When science goes wrong. The short version is that luck, the habit of publishing only results that show something, and many small choices in how data are looked at can make a false finding look solid. What matters here is the practical lesson. One published study, especially a surprising one, is weaker evidence than it looks. A finding repeated by others, in large samples planned in advance, is much stronger.

One test, many countries

The two questions meet in a project called Many Labs. Instead of one team repeating one study, dozens of laboratories run the same studies at once, on the same plan, written down and checked before any data come in. The first round ran 13 findings in 36 samples, 6,344 people in all. Ten held up clearly. Two failed completely, both claims that a brief glimpse of something, a national flag or a picture of money, quietly shifts people’s views of politics and society. Whether the people were tested in a laboratory or online, in America or elsewhere, made little difference.

Many Labs 2 went much further. It took 28 findings, some of them famous, some recent, and ran each in about 60 laboratories and online samples, 125 samples in all, with 15,305 people in 36 countries. There were 39 samples from the United States, nine from the Netherlands and six from Poland, and five from India, Nepal’s neighbor. None was from Nepal.

Fifteen of the 28 findings came out clearly again, 14 of them even by a much stricter test. The typical effect was about a quarter of the size first reported. Some held well. Asked about a runaway train, people still judged it more acceptable to switch the train onto a track where it kills one person than to push a man in front of it to save five, though the gap was about half as wide as first reported. Others vanished. One study of 27 people had found that copying out a story of sabotaging a coworker made people want cleaning products more, as if to wash away the guilt. Among 7,001 people the two groups rated the cleaning products exactly the same, 3.95 on average in each.

Then came the result that matters most for this chapter. Whether a finding held depended mostly on the finding, not on where it was tested. Of those 14, 13 held about equally strongly in the samples from WEIRD countries and in those from less WEIRD ones. A finding that failed in one place rarely came back to life in another.

One finding was itself about India. An earlier study had found that Indians were less likely to call something they had done for themselves a choice than something they had done involving other people, while for Americans the two were alike. When Many Labs 2 tested 122 students in university laboratories in India, that difference didn’t appear.

Does this mean Henrich was wrong? The Many Labs 2 team themselves were careful. Their split between WEIRD and less WEIRD was made only by country, and they called it exploratory. They said it ignored the great variety inside each group, and that a real test of WEIRDness would have to vary every one of its five parts on purpose. They also found occasional signs of cultural differences, three findings in all. And their people were not Henrich’s: most were tested in laboratories, the rest online, at an average age of about 23, while many of Henrich’s biggest gaps were with foragers, herders and farmers far from any university. So the two projects looked at different people. Findings that hold up tend to hold among the world’s students, and how far they reach beyond them is often untested.

What it means in Nepal

For a reader in Nepal, then, a finding in a textbook can be in one of three states. Some have been tested in many kinds of people and held, and you can take them as being about you. Some were found in one kind of people and failed elsewhere, or failed to repeat at all. And for many, nobody has tested them on anyone like you, so nobody knows.

Even asking the question takes care, because the tools of psychology were built in WEIRD places too. Many studies around the world use standard questionnaires to measure personality on five broad traits, such as how outgoing or how organized a person is. When researchers analyzed surveys of 94,751 people in 23 low- and middle-income countries, asked the questions face to face, the questions mostly failed to measure the traits they were meant to. The answers didn’t hang together the way the five traits predict. In online surveys from the same countries, answered by people who chose to take part, most of them with a college education, the same questions worked much better. The authors point to habits of answering, to being asked by an interviewer, and to how much schooling people have had. Among the Tsimane of Bolivia, the five traits didn’t appear either. There the answers grouped into two broad traits, which the researchers think may suit life in a small society, so the shape of personality itself may not be the same everywhere. Either way, a questionnaire made in one place can’t simply be carried to another.

A study in Nepal shows what it takes to carry a tool across. A team working with TPO Nepal, a Nepali organization in mental health, wanted a short questionnaire that health workers in Chitwan could use to find patients who might have depression. They started from one widely used in American clinics, and before using it they took it through translators, a psychologist and a psychiatrist, and discussions with people from the district, item by item, then had it translated back into English by someone who hadn’t seen the original. The English “feeling down” and the word “depressed” made no sense translated straight into Nepali, so the item was rewritten around being frustrated, losing hope and feeling unable to do anything. Feeling you had let your family down became blaming yourself and harming your family’s इज्जत, its standing. Trouble concentrating was asked about with everyday tasks, such as sorting rice or cutting grass. The original answers ask how many days in the last two weeks, and in the discussions people found they were trying to recall each day, so the answers became not at all, sometimes, usually and always, with pictures of water glasses filled to different levels.

Then they checked the result against a full interview by trained counselors, among 125 patients from rural health posts in Chitwan. Carried across this carefully, the questionnaire missed almost none of the patients who had depression, though, like such questionnaires in rich countries, it also picked out many who didn’t. The same team asked about the Nepali idea of a मन problem, a trouble of the heart-mind. It caught almost every patient with depression too, but many more without it, because a मन problem covers many kinds of distress, not only depression.

The other studies in this book from Nepal are like that: careful, small, and in a few places. The national survey of mental health, asked only in Nepali, may have badly undercounted problems in Province 2, now Madhesh, where its authors note the local language is spoken more than Nepali. Pamela Cole’s observations of how adults answered children’s feelings come from a few Tamang and Brahman villages about thirty years ago. They are good evidence about those people at that time. Each is a reason to look, not a portrait of a whole people.

So read every finding in this book with two questions in mind, the ones it keeps asking. Who was in the study? And when others did it again, did it hold? A result that held in many labs, across many countries, is about as solid as psychology gets. A result from one small study of one kind of person may be true, but nobody yet knows, and that includes many of the most famous ones. Psychology is still at the start of becoming a science of all humans, and much of the looking in places like Nepal is still to be done.

Found a mistake? Please tell us.