Final Exam Model Paper 5
Theory paper: 50 marksTime: 2 hours
Practical exam: 20 marksTime: 1 hour 30 minutes
All questions are compulsory.
A full practice run of the Grade 12 final exam: the theory paper (50 marks, 2 hours) and, below it, the practical exam (20 marks, 1 hour 30 minutes, at a computer, marked by the school). In the theory paper, work out on paper what each program does, as in the exam hall: its programs have no Run button. In the practical exam, the programs run on this page.
A practice paper by Science Playground, made to its own course’s grid; not an official exam paper.
Write your answers on paper first. Then open each answer and its marking scheme, and give yourself one mark for each point you got. In multiple choice, your first choice counts.
Section A
10 × 1 = 101. Choose the correct option for each of the following questions.
What is a probabilistic algorithm?
Pick an answer to see why.
A company offers Karnali’s education office a program that “finds the best possible timetable for every school in the province, with any number of teachers, rooms and rules, each in under a second.” Which judgment is right?
Pick an answer to see why.
In test-driven development, what comes first?
Pick an answer to see why.
Every test of a school’s fee program passes, but the coverage report shows that the lines inside if student["scholarship"]: never ran. What does this tell the team?
Pick an answer to see why.
What does it mean for copies of data to be eventually consistent?
Pick an answer to see why.
A news website keeps its articles in 20 shards by article number. On SEE results day, one article, the list of results links, gets most of the visits, and its shard is overwhelmed while the others are idle. Why would adding 20 more shards not help?
Pick an answer to see why.
A cooperative’s app answers 85% of balance-history requests from a cache, in 2 ms, and the other 15% from the database, in 200 ms. What is the average time per request?
Pick an answer to see why.
What is a replay attack?
Pick an answer to see why.
Finding a nonce that makes a hash start with one zero (in hexadecimal) takes about 16 tries on average. Why does each extra zero make the work about 16 times more?
Pick an answer to see why.
A company proposes giving every Grade 5 student in Bajura a tablet with an AI tutor. Its plan lists the benefits: better marks, modern skills, less work for teachers. Which is the strongest criticism of the plan as a conceptual design?
Pick an answer to see why.
Section B
8 × 3 = 24Write short answers to the following questions.
A ward office’s program has these three parts. For each, give its big O, with a reason, where n is the number of houses in the ward.
i.
for house in houses: print(house["number"], house["owner"])ii. (numbers is the list of house numbers, sorted.)
low, high = 0, len(numbers) - 1while low <= high: middle = (low + high) // 2 if numbers[middle] == wanted: break elif numbers[middle] < wanted: low = middle + 1 else: high = middle - 1iii.
for house in houses: for month in range(12): print(house["number"], "tax for month", month + 1)Answer
- i.O(n): one loop, one step for each house.
- ii.O(log n): each time round, the part of the list still to search is halved, so even a million houses need only about 20 rounds.
- iii.O(n), though it is a loop inside a loop: the inner loop always runs 12 times, whatever n is, so the work is 12 × n steps, and a fixed number like 12 is dropped in big O.
Marking scheme
- i. O(n), one pass over the houses.
- ii. O(log n), halving the range each time (binary search).
- iii. O(n), because the inner loop runs a fixed 12 times, not n times.
A shop’s program keeps its day’s sales like this:
total = 0 def add_sale(price, quantity): global total total = total + price * quantity print("Total so far:", total)- i.Name two side effects of
add_sale. - ii.Write a pure function
day_total(sales), wheresalesis a list of (price, quantity) pairs, that returns the day’s total. - iii.Why is your pure function easier to test?
Answer
i. It changes a global variable, total (line 5), and it prints (line 6).
ii.
def day_total(sales): """The total of a day's sales: a list of (price, quantity) pairs.""" return sum(price * quantity for price, quantity in sales) print(day_total([(120, 3), (25, 2), (450, 1)]))print(day_total([]))Output
8600iii. Its answer depends only on its argument: call it with a list worked out by hand, compare, and it is done. There is nothing to set up or reset first (such as total left over from an earlier test), and calling it twice gives the same answer.
Marking scheme
- i. Changing the global
total, and printing. - ii. A function that takes the list and returns the total, changing nothing outside itself.
- iii. Same input, same output, nothing to set up or undo, so a test is a call and a comparison.
A school library’s app needs three changes. For each, name the kind of maintenance, and say why.
- i.The app stops with an error when a book’s title has a joined letter, such as the क्ष in शिक्षा.
- ii.The school moves the app to a new server, whose newer version of Python no longer runs one of the libraries the app uses.
- iii.Students ask for an SMS two days before a book is due back.
Answer
- i.Corrective: it fixes a fault, something the app should already do but doesn’t.
- ii.Adaptive: the app is changed to fit a change around it (the new server and its Python), though nothing in the app was wrong before.
- iii.Perfective: it makes a working app better, with a feature its users asked for.
Marking scheme
- i. Corrective: fixing a fault.
- ii. Adaptive: keeping up with a change outside the app.
- iii. Perfective: improving the app with a new feature.
A wallet company’s payment server is slow for ten seconds. Its app, on 200,000 phones, tries again every second until it gets an answer.
- i.Explain how the tries again can stop the server from recovering.
- ii.What is exponential backoff?
- iii.Why is jitter added to it?
Answer
- i.Each phone keeps sending, so the slow server gets ever more requests, many of them for payments already waiting: a retry storm. The extra load keeps it slow, or makes it fail, long after the first problem is over.
- ii.Each try waits twice as long as the one before (1 s, 2 s, 4 s, 8 s…), so the load from tries again falls quickly and the server has room to recover.
- iii.Phones that failed together would otherwise try again together, at the same moments, in waves. Jitter, a small random addition to each wait, spreads their tries out.
Marking scheme
- i. Tries again add load to an already slow server, keeping it down (a retry storm).
- ii. The wait doubles after each failed try.
- iii. A random extra wait so that phones don’t try again all at once.
A business association in Banke wants to know whether shops’ sales are better this year. It phones 500 shops from the list of registered businesses, on weekday mornings; 180 answer, and 70% of them say sales are better.
- i.Explain one way the list biases the answer.
- ii.Explain one way non-response could bias it.
- iii.Suggest how the association could get a fairer answer.
Answer
- i.Small shops, street sellers and stalls that never registered are left off the list, and their year may have been quite different.
- ii.Only 180 of 500 answered. Shopkeepers who answer the phone on a weekday morning may be those with fewer customers, or those keen to talk about a good year; those too busy, or doing badly, may not answer. If those who answer differ from those who don’t, the 70% is biased, and a bigger sample of the same kind doesn’t fix it.
- iii.Choose shops at random from a fuller list (for example, by walking every market street in chosen areas, unregistered shops included), call at different times or visit in person, go back to those who didn’t answer, and report how many answered.
Marking scheme
- i. Unregistered shops are left off the list (with why that matters).
- ii. Those who answer may differ from those who don’t, so the share is biased.
- iii. A random sample from a fuller list, more ways and times to reach people, following up non-responders.
A school in Kanchanpur tried a new way of teaching fractions with one Grade 7 class of 30 students, while its other Grade 7 class was taught the usual way. In the end-of-unit test the new class scored 4 marks more on average. A teacher resampled each class 1,000 times, as the bootstrap does, and found that the middle 95% of the differences ran from −2 to +10 marks. The head teacher announces, “The new method raises marks by 4. Every class will use it from next month.” Judge this.
Answer
The evidence doesn’t support it. The 4 marks is one estimate, and its 95% interval runs from −2 to +10: it includes 0, and even differences in favor of the old way. Chance alone, in which students happened to be in which class, could give a gap of 4, so the data can’t show that the method helps (or that it doesn’t).
Besides, the two classes weren’t chosen at random: they had their own teachers, times of day and students, and any of these (a confounder) could explain the gap, so even a clear difference couldn’t be put down to the method.
A fairer test: try it in many classes, with chance deciding which classes use the new method, the test and the measure decided in advance, and report the difference with its interval. Change every class only if the interval is clearly above 0 and the gain is worth the work.
Marking scheme
- The interval includes 0 (and negative differences), so chance could give the 4 marks: not evidence the method works.
- One class each, not chosen at random: other differences (teacher, students) could explain the gap.
- A better test: many classes, chosen at random, decided in advance, reported with the interval before deciding.
For a computer club demonstration in Baitadi, Nisha and Kamal do a key exchange with p = 19 and g = 2, using numbers small enough to break by hand. Eve, listening, sees Nisha send A = 14 and Kamal send B = 15.
- i.Find Nisha’s secret a, as Eve would: by working out the powers of 2, mod 19, until one gives 14.
- ii.Use it to work out their shared key, as Eve now can.
- iii.Why can’t Eve do the same when p is a prime hundreds of digits long?
Answer
- i.The powers of 2, mod 19, each the last one doubled: 2¹ = 2, 2² = 4, 2³ = 8, 2⁴ = 16, 2⁵ = 32 ≡ 13, 2⁶ ≡ 26 ≡ 7, 2⁷ ≡ 14. So a = 7, after 7 tries.
- ii.The key is Bᵃ = 15⁷ mod 19: 15² = 225 ≡ 16 (225 − 209); 15⁴ ≡ 16² = 256 ≡ 9 (256 − 247); 15⁷ = 15⁴ × 15² × 15 ≡ 9 × 16 × 15 = 2,160 ≡ 13 (2,160 − 2,147). The shared key is 13.
- iii.Eve’s only method is trying the powers one by one (the discrete logarithm problem). With p = 19 there are at most 18 to try; with a prime hundreds of digits long there are more than there are atoms in the universe, and no fast method is known. Nisha and Kamal, who each know their own secret, need only a few hundred multiplications by repeated squaring.
Marking scheme
- i. The powers of 2 mod 19 worked out until 14: a = 7.
- ii. 15⁷ mod 19 = 13, the shared key.
- iii. Finding a means trying powers one by one (the discrete logarithm), hopeless for huge p, while the owners’ own work stays small.
What is Moravec’s paradox? Give an example. What is the difference between narrow AI and artificial general intelligence?
Answer
- Moravec’s paradox: things that are hard for people, like long arithmetic or chess, are often easy for computers, while things a small child does without thinking, like seeing, walking and picking up a cup, are very hard for them.
- Example: a program beat the world chess champion in 1997, but a robot that can walk into an unfamiliar kitchen and make a cup of tea is still very hard to build.
- Narrow AI is good at one kind of task (reading digits, translating, playing Go); artificial general intelligence would learn and do almost any task a person can.
Marking scheme
- Moravec’s paradox: hard for people can be easy for machines, and easy for people hard for machines.
- A fitting example (chess against walking, seeing or handling things).
- Narrow AI does one kind of task; AGI would do almost any task a person can.
Section C
2 × 8 = 16Write long answers to the following questions.
A municipality in Kailali links its office’s server, M, to the offices of wards 1 to 5. These are the delays along the links, in milliseconds (made-up delays). Every link works both ways.
| Link | Delay (ms) |
|---|---|
| M – Ward 1 | 2 |
| M – Ward 2 | 5 |
| Ward 1 – Ward 2 | 4 |
| Ward 1 – Ward 3 | 1 |
| Ward 2 – Ward 4 | 6 |
| Ward 2 – Ward 5 | 4 |
| Ward 3 – Ward 4 | 3 |
| Ward 3 – Ward 5 | 10 |
| Ward 4 – Ward 5 | 9 |
- i.Use Dijkstra’s algorithm to find the quickest delay from M to every ward office. Write a row for each visit: the office visited, its delay, what changes, and the offices waiting, nearest first. (3)
- ii.Write the quickest way from M to Ward 5, and its delay. (1)
- iii.A trainee suggests a quicker method: “From M, always follow the quickest link to an office you haven’t been to, until you reach Ward 5.” Follow it, and write the route and its delay. (2)
- iv.Dijkstra’s algorithm is greedy too: it always takes the nearest waiting office and never changes its mind. Why is its choice safe when the trainee’s is not? What must be true of the links? (2)
Answer
i.
| Visit | Its delay | Links checked: what changes | Waiting, nearest first |
|---|---|---|---|
| M | 0 | Ward 1 2, Ward 2 5: new | Ward 1 2, Ward 2 5 |
| Ward 1 | 2 | Ward 2 by Ward 1, 6: not shorter than 5. Ward 3 3: new | Ward 3 3, Ward 2 5 |
| Ward 3 | 3 | Ward 4 6: new. Ward 5 13: new | Ward 2 5, Ward 4 6, Ward 5 13 |
| Ward 2 | 5 | Ward 4 by Ward 2, 11: not shorter than 6. Ward 5 5 + 4 = 9: shorter than 13, now from Ward 2 | Ward 4 6, Ward 5 9 |
| Ward 4 | 6 | Ward 5 by Ward 4, 15: not shorter than 9 | Ward 5 9 |
| Ward 5 | 9 | all visited: stop |
Quickest delays: Ward 1 2 ms, Ward 3 3 ms, Ward 2 5 ms, Ward 4 6 ms, Ward 5 9 ms.
ii. 9 ms, M → Ward 2 → Ward 5 (Ward 5 came from Ward 2, Ward 2 from M).
iii. From M the quickest link is to Ward 1 (2); from Ward 1, to Ward 3 (1); from Ward 3, to Ward 4 (3, quicker than Ward 5’s 10); from Ward 4, to Ward 2 (6, quicker than Ward 5’s 9); from Ward 2, to Ward 5 (4). M → Ward 1 → Ward 3 → Ward 4 → Ward 2 → Ward 5: 2 + 1 + 3 + 6 + 4 = 16 ms, nearly twice the quickest.
iv. The trainee chooses by one link at a time, without counting the delay already spent or where the route leads, so a cheap first link (M – Ward 1) can lead far from the goal. Dijkstra’s algorithm chooses by the whole delay from M: the nearest waiting office can’t be reached more quickly some other way, because any other way passes through an office already waiting at least as far away, and more links can only add delay. That holds only if no link has a negative delay, which is true of delays, distances and fees.
Marking scheme
- i. The offices visited in order: M, Ward 1, Ward 3, Ward 2, Ward 4, Ward 5.
- i. Ward 5 relaxed from 13 to 9 when Ward 2 is visited, and Ward 2 and Ward 4 not changed by the longer ways (6 and 11, 15).
- i. The quickest delays to every office: 2, 5, 3, 6, 9 ms for wards 1 to 5.
- ii. 9 ms, by M → Ward 2 → Ward 5.
- iii. The trainee’s route M → Ward 1 → Ward 3 → Ward 4 → Ward 2 → Ward 5.
- iii. 16 ms, slower than Dijkstra’s 9 ms.
- iv. Dijkstra’s choice is by total delay from the start, so the nearest waiting office can’t be reached more quickly another way; the trainee’s looks at one link only.
- iv. It needs every link’s delay to be zero or more.
A mobile wallet company plans a model that scores every account for fraud, and freezes an account automatically when its score is above a threshold; the owner must then visit a branch with their citizenship certificate to unfreeze it. The company tested the model on 2,000 accounts, 40 of which really were used for fraud.
| Threshold | Fraud accounts caught (of 40) | Normal accounts flagged (of 1,960) |
|---|---|---|
| 0.3 | 36 | 160 |
| 0.5 | 30 | 40 |
| 0.7 | 20 | 5 |
At the threshold 0.5, split by how the owner uses the wallet:
| Fraud, caught | Fraud, missed | Normal, flagged | Normal, not flagged | |
|---|---|---|---|---|
| App on a smartphone (1,500 accounts) | 25 | 5 | 22 | 1,448 |
| Codes on a basic phone (500 accounts) | 5 | 5 | 18 | 472 |
- i.Work out the precision and recall at each threshold. (3)
- ii.At 0.5, for each group, work out the share of normal accounts flagged, and the recall. What do the numbers mean for basic-phone users? (2)
- iii.Should the company freeze accounts automatically? Argue your case: which threshold, if any, and how the model should be used. (3)
Answer
i.
- 0.3: precision 36 ÷ (36 + 160) = 36 ÷ 196 = 18%; recall 36 ÷ 40 = 90%.
- 0.5: precision 30 ÷ (30 + 40) = 30 ÷ 70 = 43%; recall 30 ÷ 40 = 75%.
- 0.7: precision 20 ÷ (20 + 5) = 20 ÷ 25 = 80%; recall 20 ÷ 40 = 50%.
ii. Smartphone app: 22 of 1,470 normal accounts flagged, 1.5%; recall 25 ÷ 30 = 83%. Basic phone: 18 of 490, 3.7%; recall 5 ÷ 10 = 50%. A basic-phone user with an honest account is about two and a half times as likely to be frozen, and their account is half as well protected from fraud: worse on both counts. Basic-phone users are often older or live far from a branch, so a freeze costs them most. The model probably learned mostly from app users’ data.
iii. One well-argued case (other conclusions earn marks with good reasons): Not automatic freezing; a check by a person, with a different threshold for each action.
- The cost of each mistake. A freeze is a punishment for an honest owner: no money for days and a trip to a branch with documents. At 0.5, 40 honest owners are frozen to catch 30 frauds (more than half the freezes are wrong); at 0.3, 160. Only at 0.7 are most flags right (80%), but then half the fraud gets through.
- Two actions, two thresholds. Above 0.7, hold the payment and call the owner at once (precision 80%); between 0.3 and 0.7, send the account to a fraud officer to look at before anything happens, or ask the owner to confirm the payment by a call or code. That catches most fraud (recall up to 90%) while few honest owners are stopped without a person checking.
- The gap first. Before using it for basic-phone users, collect and label more of their accounts and retrain, then measure by group again; until then, no account is frozen on the model alone. Owners are told why and can get a quick review by phone (appeal), the numbers are checked by group every month (fraud changes: drift), and there is a way to switch it off.
Marking scheme
- i. Precision and recall at 0.3: 18% and 90%.
- i. At 0.5: 43% and 75%.
- i. At 0.7: 80% and 50%.
- ii. Normal accounts flagged: 1.5% (app) and 3.7% (basic phone); recall 83% and 50%.
- ii. Basic-phone users are wrongly frozen more often and protected less, and a freeze costs them more.
- iii. A clear conclusion argued from the cost of a wrong freeze against a missed fraud, using the threshold table.
- iii. The basic-phone gap dealt with at its cause (more of their data, retraining, measured by group).
- iii. Safeguards: a person checks before a freeze, owners told and able to appeal, checks for drift by group, a way to stop.
Practical exam
20 marksMarked by the school. Work alone at a computer, with no internet and no AI, in front of the teacher, for 1 hour 30 minutes. Write and run each program, show it working, and answer the teacher’s questions. On this page, press Edit on a program to type in it, and Run to run it.
The computer lab on Saturday. The school’s computer lab holds one session at a time, and a session may start the moment another ends. Write a function most_sessions(requests) that returns the names of the most sessions that fit, choosing greedily by the earliest finish, and a function best_count(requests) that finds the most that fit by trying every set of requests. Write automatic tests, run by a small test runner, including one that checks most_sessions against best_count on many random lists of requests. Show the tests passing, then print the sessions chosen for Saturday and the best possible number.
REQUESTS = [ ("Coding club", 7, 9), ("Robotics club", 8, 12), ("Typing practice", 9, 10), ("Web design", 10, 12), ("Grade 6 Scratch", 11, 13), ("Chess club", 12, 14), ("Photo editing", 13, 15), ("Teacher training", 14, 16),] def most_sessions(requests): return [] def best_count(requests): return 0 # Your tests hereAnswer
import itertoolsimport random REQUESTS = [ ("Coding club", 7, 9), ("Robotics club", 8, 12), ("Typing practice", 9, 10), ("Web design", 10, 12), ("Grade 6 Scratch", 11, 13), ("Chess club", 12, 14), ("Photo editing", 13, 15), ("Teacher training", 14, 16),] def most_sessions(requests): """The most sessions that fit in the lab, one at a time: always the one that finishes earliest.""" chosen = [] free_from = None for name, start, end in sorted(requests, key=lambda r: r[2]): if free_from is None or start >= free_from: chosen.append(name) free_from = end return chosen def best_count(requests): """The most sessions that fit, found by trying every set of requests (slow, but sure).""" for size in range(len(requests), 0, -1): for group in itertools.combinations(requests, size): times = sorted((start, end) for name, start, end in group) if all(times[i][1] <= times[i + 1][0] for i in range(len(times) - 1)): return size return 0 def test_empty(): assert most_sessions([]) == [] def test_long_first_session(): # starting earliest would take the long one and fit only 1 assert most_sessions([("Long", 7, 15), ("A", 8, 9), ("B", 9, 10)]) == ["A", "B"] def test_back_to_back(): assert most_sessions([("A", 7, 8), ("B", 8, 9)]) == ["A", "B"] def test_matches_best(): random.seed(1) for trial in range(200): requests = [] for i in range(6): start = random.randint(7, 15) requests.append((str(i), start, start + random.randint(1, 3))) assert len(most_sessions(requests)) == best_count(requests), f"{requests}" passed = failed = 0for name, test in list(globals().items()): if name.startswith("test_"): try: test() print("PASS", name) passed += 1 except AssertionError as problem: print("FAIL", name, problem) failed += 1print(passed, "passed,", failed, "failed")print(most_sessions(REQUESTS))print("Best possible:", best_count(REQUESTS))Output
PASS test_emptyPASS test_long_first_sessionPASS test_back_to_backPASS test_matches_best4 passed, 0 failed['Coding club', 'Typing practice', 'Web design', 'Chess club', 'Teacher training']Best possible: 5The greedy choice fits 5 sessions, and trying every set confirms that 5 is the most. best_count is sure but slow (it may try every one of the 2ⁿ sets), so it is used only to test the fast method on small lists.
Marking scheme
- Sorts the requests by finishing time and takes each one that starts when or after the last one taken ends.
best_counttries sets of requests (largest first, withitertools.combinationsor otherwise) and checks that none overlap.- Returns the names (not prints), and
[]for no requests. - Tests written as
test_functions withassert, with answers worked out by hand. - A test where a long first session would spoil starting-earliest, and a test against
best_counton many random lists. - A runner that runs every test and reports passes and failures; all pass.
- Prints the five sessions chosen and the best possible, 5.
The top of the quiz. A teacher’s program reads the marks of a class quiz, out of 10, and finds who came top. Run the tests, find why one fails, fix the code, and run them again.
QUIZ = "Aarav,9\nBinita,10\nChirag,7\nDiksha,8" def top_student(text): """The name with the highest mark; each line is name,mark (marks out of 10).""" best_name, best_mark = None, None for line in text.split("\n"): name, mark = line.split(",") if best_mark is None or mark > best_mark: best_name, best_mark = name, mark return best_name def test_one_student(): assert top_student("Aarav,9") == "Aarav" def test_class(): assert top_student(QUIZ) == "Binita", f"gave {top_student(QUIZ)}" def test_low_marks(): assert top_student("Chirag,7\nDiksha,8") == "Diksha" passed = failed = 0for name, test in list(globals().items()): if name.startswith("test_"): try: test() print("PASS", name) passed += 1 except AssertionError as problem: print("FAIL", name, problem) failed += 1print(passed, "passed,", failed, "failed")Answer
The first run:
Output
PASS test_one_studentFAIL test_class gave AaravPASS test_low_marks2 passed, 1 failedBinita has 10, the highest, but Aarav’s 9 wins. The marks split from each line are text, not numbers, so line 9 compares them as text, character by character: "9" > "10", because the character 9 comes after the character 1. The other tests passed only because all their marks have one digit. Turn each mark into a number before comparing:
QUIZ = "Aarav,9\nBinita,10\nChirag,7\nDiksha,8" def top_student(text): """The name with the highest mark; each line is name,mark (marks out of 10).""" best_name, best_mark = None, None for line in text.split("\n"): name, mark = line.split(",") mark = int(mark) # a number, not text if best_mark is None or mark > best_mark: best_name, best_mark = name, mark return best_name def test_one_student(): assert top_student("Aarav,9") == "Aarav" def test_class(): assert top_student(QUIZ) == "Binita", f"gave {top_student(QUIZ)}" def test_low_marks(): assert top_student("Chirag,7\nDiksha,8") == "Diksha" passed = failed = 0for name, test in list(globals().items()): if name.startswith("test_"): try: test() print("PASS", name) passed += 1 except AssertionError as problem: print("FAIL", name, problem) failed += 1print(passed, "passed,", failed, "failed")Output
PASS test_one_studentPASS test_classPASS test_low_marks3 passed, 0 failedMarking scheme
- Runs the tests and names the failing one: the class gives Aarav, not Binita.
- Finds the cause: the marks are compared as text, and
"9" > "10". - Fixes the code with
int(mark)before comparing (not by changing the test). - Runs all the tests again, and all pass.
Registered twice. A ward office in Jajarkot tried a model that finds pairs of records in its senior citizens’ allowance register that are the same person registered twice (and paid twice). It was tested on 60 pairs checked by a ward secretary: truth has D where the pair really was the same person, said has D where the model said so, and script is R where the names are written in Roman letters and N where they are written in Devanagari. (The records and results are made up.)
Write a program that counts the four outcomes and works out the model’s precision and recall, for all 60 pairs and for each script apart. Then write one limit of the model, in a sentence for the ward office, with the numbers that show it.
truth = "--D--D----D--DD-DD--D----D-----D---D---D-D--D-D--D---D------"said = "-----DD------DD-D---D----D---------D--DD----D----D---D-----D"script = "NRRNRRNRNNNRRRRRRNNRNNNNRRRNRNNNRNNRNRRRRNRNRRNNRNNNRNRNNNRR" # Your program hereAnswer
truth = "--D--D----D--DD-DD--D----D-----D---D---D-D--D-D--D---D------"said = "-----DD------DD-D---D----D---------D--DD----D----D---D-----D"script = "NRRNRRNRNNNRRRRRRNNRNNNNRRRNRNNNRNNRNRRRRNRNRRNNRNNNRNRNNNRR" def measure(name, which): """Counts the outcomes for the pairs whose names are written in `which`.""" tp = fp = fn = tn = 0 for real, model, s in zip(truth, said, script): if s not in which: continue if real == "D" and model == "D": tp += 1 elif real == "D": fn += 1 elif model == "D": fp += 1 else: tn += 1 precision = tp / (tp + fp) recall = tp / (tp + fn) print(f"{name:11} TP {tp:2} FP {fp} FN {fn} TN {tn:2} precision {precision:.0%} recall {recall:.0%}") measure("All", "NR")measure("Roman", "R")measure("Devanagari", "N")Output
All TP 11 FP 3 FN 6 TN 40 precision 79% recall 65%Roman TP 8 FP 2 FN 1 TN 19 precision 80% recall 89%Devanagari TP 3 FP 1 FN 5 TN 21 precision 75% recall 38%One limit: “The model found 8 of the 9 double registrations written in Roman letters, but only 3 of the 8 written in Devanagari (38%); and it was tested on only 17 double registrations, too few to be sure.” Names in Devanagari can be spelled in more than one accepted way, which the model may not have learned. Until it does better, the secretary should check the Devanagari records by hand.
Marking scheme
- A loop over the strings together that counts true positives, false positives, false negatives and true negatives.
- Precision and recall worked out from the counts: 79% and 65% for all pairs.
- The two scripts measured apart (recall 89% for Roman letters, 38% for Devanagari).
- One limit stated in a sentence with its numbers (the Devanagari recall, or only 17 double registrations).
Viva. The teacher asks each student about their own work, for example:
- i.
best_counttries sets of requests. How many sets are there of the 8 Saturday requests, and of a whole term’s 60 requests? Why was it still worth writing? (2) - ii.In the quiz program, why did two tests pass though the code was wrong? (1)
- iii.The double-registration model is right about 51 of the 60 pairs, 85%. Why is that not enough to trust it? What would you do? (2)
Answer
- i.Each request is either in a set or not, so 8 requests have 2⁸ = 256 sets, and 60 have 2⁶⁰, about 10¹⁸: at a billion sets a second, more than 30 years. So it is useless for a real term. But it is sure, and on small lists it is quick, so it is the method known to be right that the fast greedy choice is tested against, on many random lists: the evidence that greedy really works.
- ii.All their marks have one digit, and for one-digit numbers comparing as text gives the same order as comparing as numbers. Only a test with 10 shows the bug: tests must include values where the code could go wrong (here, more digits).
- iii.Most pairs are not double registrations, so accuracy is mostly about them; it hides that 5 of the 8 Devanagari double registrations were missed, each one an allowance paid twice. Measure each script apart, test on more Devanagari pairs, find why (spellings), improve and test again, and check those records by hand meanwhile.
Marking scheme
- i. 2⁸ = 256 sets for 8 requests, and 2⁶⁰ (about 10¹⁸) for 60: far too many.
- i. It is sure, so on small lists it tests that the fast greedy method gives the best answer.
- ii. With one-digit marks, text order and number order agree, so those tests couldn’t catch it.
- iii. Double registrations are few, so a high accuracy can hide many misses (5 of 8 in Devanagari).
- iii. Measure by script, test on more Devanagari pairs, find and fix the cause, check by hand meanwhile.
Your score
0 / 50 theory
0 / 20 practical
24 questions not marked yet
| Section A | – | Full marks 10 |
| Section B | – | Full marks 24 |
| Section C | – | Full marks 16 |
| Practical exam | – | Full marks 20 |
Marks lost, by unit
Mark a question to see your score here.
