How Machines Learn

Think of a fruit seller at a collection center in the Terai in the mango season. A cart of mangoes comes in, and she sorts them: ripe ones to sell today, the rest to wait a few days. She looks at each mango’s skin and presses it gently with her thumb. In a second she knows.

Now imagine a machine to help her. A camera looks at each mango and gives its skin color as a number, from 0 (green) to 1 (yellow). A soft pad presses it and gives its softness, a score from 0 (hard as a stone) to 10 (very soft). (Both scores are our own, made up for this unit; a real machine would give its own numbers.) To teach it, she sorts a few hundred mangoes herself, and for each one writes down the machine’s measurements and her own answer: ripe, or not ripe.

Each row of Figure 10.1 is one example. The things measured about every example, here its skin color, softness and weight, are its features. The right answer for that example, the thing the model must learn to give, is its label: here, ripe or not ripe. A collection of examples like this is a dataset, and the examples a model learns from are its training data.

You have met these before without the names. In Grade 6, the fruit sorter’s features were a fruit’s size and color, and its label was kagati or suntala. In a spreadsheet (Unit 5), each example would be a row, each feature a column, and the label one more column.

Choosing the features

A model knows only its features. It never sees the mango, only the numbers. So the features must be ones that change with the label. Softness and skin color change as a mango ripens, so they help a lot. Weight depends mostly on how big the mango is, not on how ripe it is, so it helps very little. The day of the week the mango was picked would not help at all.

For photos and voices the features are numbers too: the red, green and blue of every pixel in a photo (Unit 2), or how loud a sound is, thousands of times a second. A photo has millions of these numbers. Nobody picks out the useful ones by hand; the model finds its own patterns in them, such as edges, shapes and textures. That is why it can learn things no one thought to tell it, and also why it can learn the wrong thing: if every photo of a cow among its examples was taken in a green field, it may learn “green field” as much as “cow”.

Labels come from people

Somebody has to give every example its label: the fruit seller pressing each mango, a doctor marking X-rays, people paid to name photos, as in Grade 6’s story of ImageNet. Labeling takes a long time, and people make mistakes, or don’t agree. A model can’t tell a wrong label from a right one. It learns them both.

Examples and nonexamples

To learn what “ripe” is, a model must also see what “ripe” is not. Things that are not what the model is looking for are called nonexamples: here, the unripe mangoes, which are rows of the dataset like any other, with the label not ripe. A model shown only ripe mangoes would have nothing to compare them with. It would call every mango ripe, and a stone too.

The most useful nonexamples are the ones that look a little like the examples. A model that finds momos in photos needs photos of momos, and also many photos that are not momos: plates of idli, a bowl of rice, a white cloth folded on a plate. They teach it exactly where “momo” ends.

Imagine your teacher gives you twenty math questions to practice, with their answers. A week later the test is the same twenty questions. A friend who understood none of it, but learned the twenty answers by heart, gets full marks. The test has told the teacher nothing. A good test asks new questions: like the ones practiced, but not the same.

A model must be tested the same way. Remember how a nearest-neighbor model works: it gives the label of the example closest to the new thing. Ask it about one of its own examples, and the closest example is that very one, at no distance at all. So it gives that example’s label, every time. Tested on its own training examples, it would score 100%, even if it were useless on anything new. And handling new things is the whole job.

So before training, the examples are split in two:

• the training set: the examples the model learns from;
• the test set: examples kept apart, never used in training, and used only to check the model at the end.

Often about one example in five is kept for testing, picked at random so that the test set is like everything else. After training, the model answers for each example in the test set, and its answers are compared with their labels.

Accuracy

The simplest measure of how good a model is, is its accuracy: the share of the test examples it got right, usually as a percentage. (In Unit 5, accuracy meant how close one value is to the truth; for a model, it is how often its answers are right.)

$$\text{accuracy} = \frac{\text{number right}}{\text{number tested}} \times 100\%$$

In Figure 10.2 the model got 4 of the 5 test mangoes right, so its accuracy is 4 ÷ 5 × 100% = 80%.

A bad example

In this model, the fruit seller has made one mistake. A ripe mango, yellow and soft, was written down as not ripe. Find it.

So one wrong label cost this model two right answers. Fixing the data, and not the program, made it better. That is how most models are improved in real life.

Now imagine that the collection center starts buying from a new farm, which grows Bambaiya mangoes, one of the common kinds in Nepal. A Bambaiya is different: it is often still mostly green when it is ripe. The center’s model learned only from mangoes of kinds that turn yellow as they ripen. What will it say?

A model that works well for one group and badly for another, because of how it was made, is said to be biased. Here the cause was one-sided data: training data with none, or too few, of some of the things the model will meet. It is one of the commonest causes of bias in AI. It is the same problem you met in Unit 5’s survey, where asking only one class gave answers that didn’t stand for the whole school.

Look at what the total hid. 75% sounds fairly good. Only the score for each group showed that the model was right about every other mango and wrong about half the Bambaiya. Worse, if the center had tested it only on mangoes of the other kinds, it would have scored 100%, and the center would have bought it happily. A one-sided test set gives a misleading result.

Bias you can meet

The mango model is made up, but real AI has been caught being one-sided in exactly this way, and some of it is AI you use.

Photos. In 2017, researchers at Google looked at where the photos came from in ImageNet, the famous collection from Grade 6. Of the photos whose country they could find, about 45% were from the United States. India, with far more people, had about 2%. They then tested models trained on such collections with photos of bridegrooms at weddings in India, found by people in Hyderabad. The models recognized them much less well than the bridegrooms in their usual test photos; one often called them “chain mail”, a kind of armor. The soap at the start of this unit is the same story: the 2019 study found that, for things photographed in the homes with the lowest incomes (under about US$50 a month), the systems were right about 10% less often than in the homes with the highest (over US$3,500).

A model that learned what a house looks like mostly from photos of city streets, concrete houses several stories tall, may not see a house at all in a village like the one below.

Voices. Voice typing and voice assistants learn from recordings of people speaking, with what they said written out. Far more such recordings exist in English than in Nepali, and fewer still in Maithili, Bhojpuri, Tamang or Nepal Bhasa. So voice AI understands English best, and Nepali less well. Even within Nepali, a model that learned mostly from people in Kathmandu may hear a speaker from Jumla or Janakpur less well: their accents were rare among its examples.

Faces. Face unlock on a phone, and the cameras that check faces at some offices and airports, use models trained on photos of faces. If most of those faces had light skin, the model works worse for people with darker skin.

Nobody decides to build a biased model. Bias gets in through ordinary decisions that people make at each step of working with data, often without noticing. At each step, there is a question to ask.

1. Collecting the data. Who and what is in it, and who is missing? (Only city photos; only Kathmandu voices; only kinds of mango that turn yellow.) Ask: does the data include every kind of person and thing the model will be used for, in fair numbers?
2. Labeling it. Who gives the right answers, and could they be wrong or one-sided? (A tired labeler mixing up two leaf diseases; labelers who think a wedding dress must be white.) Ask: who labeled it, how were the labels checked, and do they agree?
3. Choosing the features. Does a feature stand in for something unfair? (“Has internet at home” as a feature for a scholarship says more about a family’s money than about the student.) Ask: could this feature treat a group unfairly?
4. Testing. Is the test set like the real world, and is the score given for each group? (The mango model tested only on yellow kinds.) Ask: was it tested on everyone it will be used on, group by group?
5. Using the model. Is it being used on people or things it was never trained for? (A model made in another country, used unchanged in Nepal.) Ask: is it being used where its data came from?

The same goes for conclusions drawn from data without any AI at all. A survey that only asked people with a phone, or a total that hides a group, can lead to a wrong conclusion just as easily as to a wrong model.

You fixed two models on this page by changing their data: taking away a wrong label, and adding the missing Bambaiya. People who build AI improve models the same way. A good routine:

1. Test, group by group. Find the accuracy for each group of people or things the model will meet, not only the total.
2. Look at the mistakes. What do the ✗ have in common? Are they all one kind? Are they near an example with a strange label?
3. Fix wrong labels. Check the examples near the mistakes, and correct or take away the bad ones.
4. Collect what is missing. Add examples of the group it does badly on, and nonexamples that look like them: for the Bambaiya, ripe ones and unripe ones, so the model learns what tells them apart (softness, not color).
5. Keep it balanced. More of what the model already knows doesn’t help, as you saw. Aim for every group to be well covered.
6. Test again, on examples the model has never seen, and record the results before and after, group by group, so you know the change helped and broke nothing else.

Sometimes data can’t fix a model. If good examples can’t be collected, or if a mistake would be very serious, the job may be better left to people, or the model’s answers checked by a person before anything is decided (Grade 6’s bank). And it helps to have people from many places, and with many backgrounds, building and testing AI: they notice who is missing.