Why this matters
Ghanaians speak Dagbani, Twi, Ewe, Ga, Dagaare, Gonja and many other languages every day, yet most language technology works best in English. Speech and text tools in local languages could let a farmer in Karaga ask for weather advice by voice, or let a health worker hear guidance in a patient's own language. At least one Ghanaian agritech employer has advertised for engineers to build speech recognition and language models for exactly these low resource languages. It is hard work, and valuable.
What "low resource" means
Language models learn from huge amounts of text and recorded speech. English has an enormous supply online. Most Ghanaian languages have far less written text, few labelled datasets and limited recorded speech with transcripts. Other complications:
- Spelling varies. People write the same word in different ways, especially in WhatsApp messages and social media.
- Special letters and tone. Letters such as ɛ and ɔ appear in Twi, Ewe and Dagbani writing, but many keyboards drop them, so the same word appears with and without them.
- Code switching. People mix English and a local language in one sentence, which confuses models trained on one language at a time.
Turning text into numbers
Models need numbers, so text is first split into pieces called tokens: whole words, parts of words or single characters. Each token gets an ID number, and later a vector. For languages with little data, splitting into often works better than whole words, because it copes with spelling variation and rare words.