Your Language Decides What the Model Costs You

Language models do not read characters or words; they read tokens, produced by a segmenter trained on a text corpus. That corpus determines which sequences get a compact representation. Text resembling the training distribution is packed efficiently, often a token per common word. Text that does not is broken into fragments — sometimes into individual characters, sometimes into pieces that cut across meaningful units. Since English dominates the corpora these segmenters are built from, English is the language the compression was optimised for. The measurable effect is that an identical passage translated into different languages produces very different token counts. Languages that share the Latin alphabet and a lot of vocabulary with English pay a modest premium. Languages with rich inflection, extensive compounding, or a non-Latin script can pay several times more for the same content. Since both price and context limits are denominated in tokens, this is simultaneously a cost multiplier and a capacity reduction: the same document consumes a larger share of the available window, so less can be supplied as context before something must be dropped. What makes this awkward rather than merely unfair is that the penalty compounds with a quality gap in the same direction. The languages segmented inefficiently are generally the ones with less training data, so they receive both the higher price and the weaker output. A team building for such a market pays more per request and gets less for it, and neither half of that appears anywhere in a product comparison. It is also invisible in most internal reporting, because costs are usually aggregated by feature rather than by language, so nobody sees that one market's unit economics differ from another's by a factor that would change the decision. There are practical responses, though none of them is a fix. Measure token counts per language on your own representative content rather than trusting a general rule, since the ratio varies enormously by domain and by how much of the text is proper nouns, numbers or code. Budget context windows against the worst case rather than the average, or a retrieval step that fits comfortably in one language will silently truncate in another. Compare models on tokenisation efficiency for the languages you actually serve, because segmenters differ substantially and the difference is rarely mentioned in any announcement. And when evaluating quality, evaluate in the target language: a system judged on English and deployed in another is being measured on the case it handles best.