How Does Romanized Nepali Typing Work?: Computing Further 9–12

You want to text your friend नमस्ते, but your phone shows English letters. So you switch on a Nepali keyboard that lets you type what you hear: n a m a s t e. The word नमस्ते appears. Seven English letters went in, and six Nepali codes were stored. Something on the phone cut your letters into pieces and swapped each piece for Nepali.

That something is a transliterator: a program that turns text written in one script into another script, by sound. The one below follows a small table of rules. Type a word, or pick one, and tap a piece to see how it was found.

Type in small letters (capitals mean something special, as you will see). Each tile is one piece of what you typed, with what it adds to the Nepali.

Type in Roman letters:

नमस्ते

Letters typed: 7
Pieces: 7
Code points stored: 6

7 letters, 7 pieces: नमस्ते, stored as 6 code points. Tap a piece to see how it was found.

At letter 1 the program looked up “nam” ✗, “na” ✗, “n” ✓. “n” is the consonant न. It says “na” by itself until a vowel sign or a halant changes it.

At letter 2 the program looked up “ama” ✗, “am” ✗, “a” ✓. “a” after a consonant adds nothing: the consonant already says “a”.

At letter 3 the program looked up “mas” ✗, “ma” ✗, “m” ✓. “m” is the consonant म. It says “ma” by itself until a vowel sign or a halant changes it.

At letter 4 the program looked up “ast” ✗, “as” ✗, “a” ✓. “a” after a consonant adds nothing: the consonant already says “a”.

At letter 5 the program looked up “ste” ✗, “st” ✗, “s” ✓. “s” is the consonant स. It says “sa” by itself until a vowel sign or a halant changes it.

At letter 6 the program looked up “te” ✗, “t” ✓. “t” is त. A consonant came just before it, so a halant ◌् goes first and joins the two: ◌्त.

At letter 7 the program looked up “e” ✓. “e” after a consonant is its vowel sign ◌े, stored after the consonant.

16 letters, 16 pieces: क्ष्क्ष्क्ष्क्ष्क्ष्क्ष्, stored as 32 code points. Tap a piece to see how it was found.

At letter 16 the program looked up “chh” ✗, “ch” ✗, “c” ✗. “ch” is च. A consonant came just before it, so a halant ◌् goes first and joins the two: ◌्च.

At letter 16 the program looked up “shh” ✗, “sh” ✗, “s” ✓. “sh” is the consonant ष. It says “sha” by itself until a vowel sign or a halant changes it.

Before you try it: the table has t, h and th in it. What do you think tha becomes?

Pick an answer to see why.

You need lists, strings and dictionaries in Python (Grade 9). Everything about how Nepali is stored is explained here.

Try it

Type shikshaa and watch the tiles. The program found sh, then i, then ksh (three letters, the longest piece in its table), then aa. Now type kh and chh: one tile each.

The order the program looks in matters. Switch the widget below to One letter at a time, and type tha, kh and bhaktapur again.

Compare the two ways of cutting the same letters.

Type in Roman letters:

थ

Letters typed: 3
Pieces: 2
Code points stored: 1

3 letters, 2 pieces: थ, stored as 1 code point. Tap a piece to see how it was found.

At letter 1 the program looked up “tha” ✗, “th” ✓. “th” is the consonant थ. It says “tha” by itself until a vowel sign or a halant changes it.

At letter 3 the program looked up “a” ✓. “a” after a consonant adds nothing: the consonant already says “a”.

16 letters, 16 pieces: क्ष्क्ष्क्ष्क्ष्क्ष्क्ष्, stored as 32 code points. Tap a piece to see how it was found.

At letter 16 the program looked up “chh” ✗, “ch” ✗, “c” ✗. “ch” is च. A consonant came just before it, so a halant ◌् goes first and joins the two: ◌्च.

At letter 16 the program looked up “shh” ✗, “sh” ✗, “s” ✓. “sh” is the consonant ष. It says “sha” by itself until a vowel sign or a halant changes it.

Now type nepal in the first widget.

nepal comes out as नेपल, not नेपाल. Why?

Pick an answer to see why.

How it works

What the phone stores

A transliterator’s output is Unicode, the same codes you met in Grade 7 and Grade 9 (How Nepali is stored, Characters and Unicode). Three facts about Devanagari in Unicode decide what it must produce:

Every consonant carries a vowel inside it. Unicode’s own description says that क stands for /ka/, not only /k/. This built-in vowel is called the inherent vowel.
A vowel sign (मात्रा), such as े or ा, replaces the inherent vowel. It is stored after its consonant, in the order you say it, even when it is drawn on the left (कि is क then ि).
The halant (◌्) takes the inherent vowel away. A consonant, a halant and another consonant make a conjunct: क, ◌्, ष is क्ष. Unicode has no single code for क्ष or ज्ञ. They are drawn as one shape by the font, but stored as three codes each.

Pick क्ष: three code points. Then त्ह: also three codes, which a font draws as a half त joined to ह. It is valid Unicode, but not a Nepali word.

Type some text:

क U+0915 ◌् U+094D ष U+0937

Characters (numbers stored): 3

Type something, or pick one of the buttons.

Every English letter, digit, space and punctuation mark has its own number. A capital and a small letter have different numbers.

Every character, of every script, is stored as its own number.

A Nepali syllable is stored as several numbers: a letter, then its vowel sign or a halant (्) and the next letter.

Each English character is a number under 128, so it fits in one byte.

English characters take 1 byte each, but every Devanagari letter or sign takes 3 bytes: its number is over 2,047.

An emoji’s number is over 65,535, so it takes 4 bytes. Nepali letters take 3, English ones 1.

The bigger a character’s number, the more bytes it takes: 1, 2, 3 or 4.

So typing is a translation from one list of sounds to another: Roman letters in, Unicode code points out. Notice that shikshaa is 8 letters and शिक्षा is 6 code points: sh and aa are two letters for one code each, while ksh is three letters for three codes.

Longest piece first

The program’s table is a Python dictionary: each key is a piece of Roman letters, each value what it gives in Nepali. Most pieces are one letter, some two (kh, th, aa, sh), a few three (chh, ksh). Capital letters are the retroflex sounds, said with the tongue curled back: T is ट, D ड, N ण, Sh ष. Small t is त.

The program reads the typed text from left to right. At each place it tries the longest pieces the table has, 3 letters, then 2, then 1, and takes the first one it finds. This rule is called longest match first (or maximal munch), and many programs that cut text into pieces use it, from transliterators to the part of Python that reads your programs: it reads **= as one piece, not as *, * and =.

Putting the pieces together

Once the text is cut into pieces, four rules turn them into Unicode, in order:

a.A consonant after another consonant gets a halant before it, so the two join: s then t is स्त.
b.A vowel after a consonant gives its vowel sign (e gives े), and a gives nothing at all.
c.A vowel anywhere else gives the vowel letter itself: aama begins with आ.
d.A consonant at the end of a word stays as it is, with its inherent vowel. Nepali writes नेपाल, not नेपाल्, though nobody says the last a.

Rule d. is a choice. Many words do end in a halant: छन् (they are), सन् (the year), विद्युत् (electricity). Our rules can’t type those; a real keyboard has a key for the halant, or learns them as words.

Using the four rules, what does ramro become?

Pick an answer to see why.

Program it

You will write the transliterator in two steps, test it on twenty words as people really type them, and then teach it to choose with a list of words.

Step 1: the table, and longest match first

pieces(text) gives back the pieces, longest first. The for loop tries 3, 2 and 1 letters; break stops it at the first piece in the table. If no piece fits (a space, or a letter like x that isn’t in the table), the loop ends with one letter, and that letter is kept as it is. Notice i = i + len(piece): near the end of the text, text[i:i + 3] may hold fewer than 3 letters.

VOWELS = {'a': ('अ', ''), 'aa': ('आ', 'ा'), 'i': ('इ', 'ि'), 'ee': ('ई', 'ी'),          'u': ('उ', 'ु'), 'oo': ('ऊ', 'ू'), 'e': ('ए', 'े'), 'ai': ('ऐ', 'ै'),          'o': ('ओ', 'ो'), 'au': ('औ', 'ौ')}CONSONANTS = {'k': 'क', 'kh': 'ख', 'g': 'ग', 'gh': 'घ', 'ch': 'च', 'chh': 'छ',              'j': 'ज', 'jh': 'झ', 'T': 'ट', 'Th': 'ठ', 'D': 'ड', 'Dh': 'ढ',              'N': 'ण', 't': 'त', 'th': 'थ', 'd': 'द', 'dh': 'ध', 'n': 'न',              'p': 'प', 'ph': 'फ', 'f': 'फ', 'b': 'ब', 'bh': 'भ', 'm': 'म',              'y': 'य', 'r': 'र', 'l': 'ल', 'w': 'व', 'v': 'व', 'sh': 'श',              'Sh': 'ष', 's': 'स', 'h': 'ह', 'ksh': 'क्ष', 'gy': 'ज्ञ'}SIGNS = {'M': 'ं', '~': 'ँ'}LONGEST = 3 def in_table(piece):    return piece in VOWELS or piece in CONSONANTS or piece in SIGNS def pieces(text):    found = []    i = 0    while i < len(text):        for size in range(LONGEST, 0, -1):            piece = text[i:i + size]            if in_table(piece):                break        found.append(piece)        i = i + len(piece)    return found print(pieces('namaste'))print(pieces('shiksha'))print(pieces('tha'))

Output

['n', 'a', 'm', 'a', 's', 't', 'e']['sh', 'i', 'ksh', 'a']['th', 'a']

Step 2: the four rules

join(parts) puts the pieces together by the four rules, remembering in after_consonant whether the last piece was a consonant. transliterate(text) does both steps.

VOWELS = {'a': ('अ', ''), 'aa': ('आ', 'ा'), 'i': ('इ', 'ि'), 'ee': ('ई', 'ी'),          'u': ('उ', 'ु'), 'oo': ('ऊ', 'ू'), 'e': ('ए', 'े'), 'ai': ('ऐ', 'ै'),          'o': ('ओ', 'ो'), 'au': ('औ', 'ौ')}CONSONANTS = {'k': 'क', 'kh': 'ख', 'g': 'ग', 'gh': 'घ', 'ch': 'च', 'chh': 'छ',              'j': 'ज', 'jh': 'झ', 'T': 'ट', 'Th': 'ठ', 'D': 'ड', 'Dh': 'ढ',              'N': 'ण', 't': 'त', 'th': 'थ', 'd': 'द', 'dh': 'ध', 'n': 'न',              'p': 'प', 'ph': 'फ', 'f': 'फ', 'b': 'ब', 'bh': 'भ', 'm': 'म',              'y': 'य', 'r': 'र', 'l': 'ल', 'w': 'व', 'v': 'व', 'sh': 'श',              'Sh': 'ष', 's': 'स', 'h': 'ह', 'ksh': 'क्ष', 'gy': 'ज्ञ'}SIGNS = {'M': 'ं', '~': 'ँ'}LONGEST = 3 def in_table(piece):    return piece in VOWELS or piece in CONSONANTS or piece in SIGNS def pieces(text):    found = []    i = 0    while i < len(text):        for size in range(LONGEST, 0, -1):            piece = text[i:i + size]            if in_table(piece):                break        found.append(piece)        i = i + len(piece)    return found def join(parts):    out = ''    after_consonant = False    for piece in parts:        if piece in CONSONANTS:            if after_consonant:                out = out + '्'            out = out + CONSONANTS[piece]            after_consonant = True        elif piece in VOWELS:            letter, sign = VOWELS[piece]            if after_consonant:                out = out + sign            else:                out = out + letter            after_consonant = False        else:            out = out + SIGNS.get(piece, piece)            after_consonant = False    return out def transliterate(text):    return join(pieces(text)) for word in ['namaste', 'bhaktapur', 'shikshaa', 'nepal']:    nepali = transliterate(word)    print(word, '→', nepali, len(nepali), 'code points')

Output

namaste → नमस्ते 6 code pointsbhaktapur → भक्तपुर 7 code pointsshikshaa → शिक्षा 6 code pointsnepal → नेपल 4 code points

The board below takes transliterate from the program just above, as you have it (press Edit above to change it), and calls it with whatever you type. Change a rule, say, leave out the halant, and see what happens to namaste.

The tiles and the big word are the page’s own program. Your program’s answer is under them.

Type in Roman letters:

नमस्ते

Letters typed: 7
Pieces: 7
Code points stored: 6