Exam practice

• Weight: a number that says how much one input counts towards a neuron’s weighted sum, and which way (positive for, negative against).
• Bias: a number added to a neuron’s weighted sum, whatever the inputs; it sets how easily the neuron says yes.
• Activation function: the rule that turns a neuron’s weighted sum into its output, such as the step, the sigmoid or ReLU.
• Hidden layer: a layer of neurons between the inputs and the output layer; its outputs feed the next layer and are not the network’s answer.
• Forward pass: working out a network’s output from its inputs, layer by layer.
• Learning rate: the number that sets how big each step of gradient descent is: the change in a weight is the learning rate times its slope.
• Epoch: one pass of training through every training example.

The rule: a discount only for spending over Rs 2,000, and then only if the customer is a regular or pays by phone (or both). Spending is needed in every case; regular and phone each help, and either is enough on top of spending.

A neuron with two inputs says 1 where w₁ x₁ + w₂ x₂ + bias is above 0: on one side of a straight line across the square of inputs. XOR needs 1 at (0, 1) and (1, 0), opposite corners, and 0 at (0, 0) and (1, 1), the other two corners. Suppose some weights worked. The point halfway between (0, 1) and (1, 0) is (0.5, 0.5); since the weighted sum changes evenly along a straight line, the sum there is the average of the sums at those two corners, so it is above 0. But (0.5, 0.5) is also halfway between (0, 0) and (1, 1), whose sums are both 0 or below, so the sum there must be 0 or below. It can’t be both, so no weights work.
With two layers: h₁ = OR (weights 1, 1, bias −0.5) and h₂ = NAND (weights −1, −1, bias 1.5), each drawing one line; the output neuron is AND (weights 1, 1, bias −1.5) of h₁ and h₂. It says 1 only in the strip between the two lines, which holds exactly (0, 1) and (1, 0). (The drawing: the unit square with its four corners, the line x₁ + x₂ = 0.5 near (0, 0), the line x₁ + x₂ = 1.5 near (1, 1), and the strip between them shaded.)

h₁: 1 × 1 + 0 × 1 − 0.5 = 0.5, above 0, so h₁ = 1.
h₂: 1 × (−1) + 0 × (−1) + 1.5 = 0.5, above 0, so h₂ = 1.
Output: 1 × 1 + 1 × 1 − 1.5 = 0.5, above 0, so the output is 1. XOR of 1 and 0 is 1.

Inputs: 28 × 28 = 784.
With 64 hidden neurons: 784 × 64 + 64 = 50,240 for the hidden layer, 64 × 10 + 10 = 650 for the outputs: 50,890 parameters.
With 128: 784 × 128 + 128 = 100,480, plus 128 × 10 + 10 = 1,290: 101,770.
The one with 128 hidden neurons is more likely to overfit: with twice as many parameters it can memorize 500 examples even more easily, instead of learning what they have in common.

w = 100 − 0.05 × (−805) = 100 + 40.25 = Rs 140.25.
Each step is the learning rate times the slope. Near the bottom of the bowl the slope is small (at the very bottom it is 0), so the steps get smaller by themselves, and the model settles at the bottom instead of jumping past it.

Imagine you are on a hillside in thick fog and want to reach the bottom of the valley. You can’t see it, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel the slope again, and take another, and keep going.

• The hill is the error: its height at any place is how wrong the network is with those weights.
• Your position is the network’s current weights.
• The slope under your feet is how the error changes as each weight changes (the gradient).
• The size of your step is the learning rate times the slope: big steps where it is steep, small near the bottom. Too big a learning rate is like jumping: you can leap over the valley to the other side.

Each step is one small correction; training is thousands of them.

1. The 100% is very likely training accuracy, measured on the same 100 digits it learned from. With so few examples a network can memorize them all (overfitting), so it says nothing about new digits.
2. 100 examples, 10 of each digit, is far too few to represent how people write, and one test on them can’t be trusted.

Measure it properly: keep a separate test set of digits it never trained on, written by other people, ideally a few hundred; report the test accuracy with the number of test digits (“74% on 500 test digits”); and look at the mistakes. If settings are tuned, tune them on a validation set, not the test set.

The chances only say how much the drawing looks like the network’s training examples of each digit, measured in the network’s own way (where the ink is). If this 1 was drawn with a long flag and a base, its ink may lie where the training 7s had ink, so the 7 output gets a high chance. The network has no idea of “I haven’t seen anything like this”: softmax always shares out 100% between the ten digits, so it always gives an answer, often a confident one.
So a high chance is not proof of a right answer. Where mistakes matter (money on a cheque, a medical scan, a face matched by police), the network’s answer should be checked by a person, especially for unusual inputs, and the system tested on the people who will really use it.

• Source: clear (one school, practice sheets), but small and from one place.
• Quality: probably good labels, since the teachers wrote known digits; but practice sheets are written slowly and neatly, unlike marks written quickly on hundreds of answer sheets.
• Representativeness: poor. 20 teachers at one school in Lalitpur don’t represent the thousands of teachers across Nepal, of different ages and schooling, writing in a hurry, with different pens and on different paper. 5,000 digits from 20 people is 250 from each: the network may learn those 20 styles very well and others badly.
• Bias: teachers whose styles are unlike these 20 (from other regions, older or younger, used to Devanagari or Latin digits) may have their marks misread more often, which could change students’ results: a serious harm.
• Privacy: handwriting can identify a person; the teachers should agree, and the data should be stored safely and used only for this.

Better: collect real marks (or practice ones written at normal speed) from many teachers in every province, in both scripts, with their consent; keep a test set from teachers not in the training set; and measure accuracy for each group. And keep a person checking any mark the network is unsure of.

• i.A neural network. Handwriting has no rule anyone can write down, the pictures have many pixels, and the bank has huge numbers of labeled slips to train on. A network answers quickly without keeping every example. A person should still check slips it is unsure of, since money is involved.
• ii.Rules. The rule is known exactly: fee = days late × Rs 5. A one-line program gets it right every time and anyone can check it; learning it from examples would only make mistakes.
• iii.Nearest neighbors. A small table: four measurements, 200 samples. Nearest neighbors needs no training, works well at this size, and can show the samples each guess was based on, which the tea board asked for. A network would need more data and couldn’t easily explain itself.

Not in general. On this small set (2,000 small, tidied pictures), nearest neighbors was more accurate, and it is simpler and can explain itself, so here it is the better choice. But one result on one small task doesn’t make a rule. With many more examples, things change: nearest neighbors must keep every example and compare each new picture with all of them, so it gets bigger and slower as the data grows, while a network stays the same size and speed. And with tens of thousands of examples, networks, especially convolutional networks designed for pictures, become more accurate than nearest neighbors too. The right choice depends on the task, the amount of data, the need for speed and the need to explain.