AI models can need 1.5 to 5 times more tokens for some of the internet’s 20 most popular languages than for others, and up to 17 times more across 200 languages, which raises the cost of using AI in them, according to a September study by the Coalition for Language Equity in AI Research, reported by Ananya Bhattacharya at Rest of World. Script mattered more than how widely a language is spoken, so Burmese, Tibetan, Khmer, and Lao fared worst in nearly every model, while Asturian, a small language written in Latin script, came out fine. More tokens also mean slower responses and less room in the context window for documents and chat history. Maya Research CEO Dheemanth Reddy said falling token prices will ease the cost, but closing the gap will take tokenizers built for these scripts.
