The Tokeniser Tax: Why Indic Prompts Cost 8x More

What the tokeniser tax costs you

Every piece of cost-optimisation advice you have read — cache the prefix, route the easy requests to a smaller model, compress the context, batch the offline work — rests on an unstated assumption. It assumes that a token is a token: a fixed, neutral quantum of work, so that reducing the count reduces the cost proportionally and everyone measuring in tokens is measuring the same thing.

For English that assumption is close enough to true to be harmless. For Devanagari, Tamil, Bengali and Malayalam it is not true at all, and the error is not a rounding error. It is a multiplier.

The clearest measurement to date comes from a paper published on arXiv in July 2026 under the title The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages (arXiv:2607.24276, 27 July 2026). The authors measured tokenisation fertility across six widely used tokenisers and fourteen languages using the FLORES-200 parallel corpus — which matters, because a parallel corpus means every language is expressing genuinely equivalent content rather than merely similar content. Under cl100k_base, the tokeniser used by GPT-3.5 and GPT-4, Indian languages carried an average tokenisation tax of 8.0 times relative to English, reaching 13.0 times for Malayalam.

Three consequences follow, and only the first one appears on an invoice.

Your per-request cost is multiplied before you change anything. A three-times fertility multiplier means every request in that language costs three times what the identical meaning costs in English, at the same rate card, on the same model, with the same prompt design. No amount of routing or batching recovers that, because it is upstream of both.

Your effective context window shrinks. The same study found the usable context reduced to as little as 12 per cent of what an English user gets for equivalent semantic content. A nominal 200,000-token window is not 200,000 tokens of meaning; it is 200,000 tokens of encoding, and how much meaning fits in those depends on the script.

Your prompt cache holds a third as much useful text. Caches are sized in tokens. If your text is three times more expensive to encode, a cache entry of a given token size holds a third of the material. Every downstream optimisation compounds with the fertility multiplier rather than replacing it.

This is why fertility has to be measured first, not last. If you build your unit economics model on English traffic and then extend it to a multilingual product, every number in it is wrong by a factor you have not measured.

What the tokeniser tax actually is

Start with the unit. Fertility is the number of tokens a tokeniser produces per unit of text — most commonly tokens per word, sometimes tokens per character. Lower is better: a fertility of 1.0 means one token per word, which is close to ideal compression. A fertility of 12 means the tokeniser is chopping each word into a dozen pieces before the model has seen anything.

Tokens per word and tokens per character are both worth computing, and they answer different questions. Tokens per word is the intuitive figure and the one most papers report, but word boundaries are a Latin-script convention that translates awkwardly to scripts with different orthographic conventions and to agglutinative morphology, where a single written word carries what English would express in a clause. Tokens per character is scruffier as a headline but harder to game, because character counts are not a matter of convention. Report both, and be suspicious when they disagree.

Why subword tokenisers fragment non-Latin scripts

Modern tokenisers are byte-pair encoders. The training procedure is simple: start with individual bytes, count which adjacent pairs occur most often in a training corpus, merge the most frequent pair into a new token, and repeat until you have the vocabulary size you wanted. The output is a merge table — an ordered list of pair-joins that the tokeniser applies at inference time.

Everything about that procedure is corpus-dependent. The merges that get learned are exactly the merges that would have compressed the training data, and nothing else. When the training corpus is overwhelmingly English — which the paper identifies as the root cause, describing these tokenisers as trained predominantly on English-centric corpora — the merge table learns English morphology in detail and learns almost nothing about anything else.

What happens to text the merge table does not cover is the interesting part. The tokeniser does not fail; it falls back. Unmatched text decomposes towards its underlying bytes. Indic scripts are encoded in UTF-8 at three bytes per code point, and they make heavy use of combining marks — vowel signs, virama, conjunct forms — so a single visually indivisible cluster may be several code points and therefore many bytes. Without merges to reassemble them, those bytes become individual tokens. This is the mechanism the paper names directly: failed byte-pair merges that leave text fragmented into single-byte tokens.

What the correlation tells you

The paper's most useful contribution is not the size of the tax but the explanation of it. Merge failure correlates with the size of the tax at a Pearson r of 0.89 — a strong relationship, and one that rules out a lot of intuitive alternative explanations.

In particular, it means the tax is not primarily about morphological complexity, script exoticism, or how "hard" a language is in any linguistic sense. It is about whether the merge table happens to cover the script. That is an engineering property of the tokeniser, chosen when it was trained, and it is therefore fixable by engineering. A language does not have an inherent token cost. It has a token cost under a particular tokeniser, and a different tokeniser gives a different answer — which is exactly what the Indic-first tokenisers demonstrate later in this article.

How big the tax actually is

Two separate sets of published measurements are worth having in front of you. They are presented separately here on purpose, because they use different tokenisers and different units, and averaging across them would be meaningless.

The first is tokens per word under the Llama-3.1 tokeniser.

Language	Tokens per word	Ratio to English
English	1.24	1.0x
Hindi	2.67	approx. 2.2x
Tamil	12.39	approx. 10.0x

Those three tokens-per-word figures are reported measurements on the Llama-3.1 tokeniser. The ratio column is plain arithmetic on the three numbers in the same column, not a separately measured quantity. These figures are not comparable with the table below.

The second set is the cl100k_base study.

Measure	Finding
Average tokenisation tax, Indian languages vs English	8.0x
Worst measured language (Malayalam)	13.0x
Effective context window remaining, worst case	as little as 12%
Correlation of failed merges with the tax	Pearson r = 0.89
Study scope	6 tokenisers, 14 languages, FLORES-200

All five rows above come from arXiv:2607.24276, published 27 July 2026, and the multipliers specifically describe cl100k_base, the tokeniser used by GPT-3.5 and GPT-4. As of September 2026 these are the best public numbers available for this question, but they describe the tokenisers the authors tested at the time they tested them, which is not necessarily the tokeniser you are billed against.

Turning multipliers into money

The arithmetic is uncomfortable precisely because it is so simple. If an English-language workload of a thousand support requests costs your Bengaluru team ₹1,000 at whatever rate you have contracted, the same thousand requests in Malayalam under cl100k_base would cost roughly ₹13,000 at the 13.0 times figure. If a London team runs a £400-a-month English-only summarisation workload and extends it to Indian-language content at the 8.0 times average, that line becomes something in the region of £3,200.

Notice what is not in either example: a model change, a prompt change, or a quality change. The multiplier arrives entirely from the encoding.

It is also worth being precise about why falling prices do not rescue you here. Reported analysis of 2.4 billion enterprise API calls put blended token costs down 67 per cent between the first quarter of 2025 and the first quarter of 2026, from 18.40 to 6.07 US dollars per million tokens. That is a real and substantial fall. It is also completely orthogonal to this problem. A rate cut applies to every workload equally; fertility is a multiplier on volume, not on rate. After a 67 per cent price fall your Malayalam workload is cheaper in absolute terms and exactly as expensive relative to your English workload as it was before. The gap does not close because someone else lowered a price.

The three places it hurts that are not the invoice

If the tax were purely financial you could treat it as a line item and move on. It is not, and the non-financial consequences are the ones that produce confusing bug reports.

Effective context collapses

A context window is denominated in tokens, so a fertility multiplier eats it directly. The study puts the effective window at as little as 12 per cent of the English equivalent for the same semantic content, which is the difference between a document fitting comfortably and a document being silently truncated.

This produces failure modes that look like model quality problems and are not. Retrieval that returns the right passages but drops half of them at assembly. Conversations that lose the beginning of the session much sooner than your English testing suggested. Chunking configured in tokens that yields semantically tiny chunks in one language and sensible ones in another. If your multilingual retrieval pipeline uses a fixed token budget across languages, it is not treating those languages equally — it is giving some of them far less material to work with, and your evaluation will attribute the resulting weakness to the model.

Prompt caching stores less per entry

Prefix caching is one of the highest-return optimisations available, and it is sized in tokens like everything else. A cache entry that holds a full system prompt, a few worked examples and a policy document in English may hold only the system prompt in a high-fertility script. Worse, providers typically enforce a minimum token count before a prefix becomes cacheable at all, which high-fertility text crosses trivially and low-fertility text sometimes does not — so caching behaviour differs by language in ways that are invisible unless you segment your cache-hit metrics by language. If you have done the work described in the cache, route and compress playbook, segment those results by language before you report the saving. The blended figure will flatter the languages that needed the help least.

Latency and time-to-first-token rise

Prefill is proportional to input token count. More input tokens means more prefill compute, which means a longer wait before the first output token appears. The user has not written a longer message; the encoder has produced a longer sequence from it. The perceptual result is that your product feels slower in one language than another for reasons no one on the team can explain, and that latency difference falls on exactly the users you were trying to serve better by supporting their language in the first place. If you route on latency budgets — and cascade routing often does — those budgets are language-dependent whether or not you modelled them that way.

Measure it on your own corpus first

Published figures tell you the effect exists and roughly how large it can be. They do not tell you your number. Your number depends on your tokeniser, your actual traffic mix, how much of your prompt is English scaffolding versus user content, and how much boilerplate you send. Measuring it is an afternoon of work and it is always the first rung on the ladder.

The method has four steps. Sample real production traffic rather than a translated test set, because translated test sets are written in a register that real users do not use. Label each sample with a language. Run every sample through every tokeniser you are billed against. Report the median tokens per word and tokens per character per language, plus the ratio to your English baseline.

# fertility.py — measure tokens per word on YOUR traffic, by language.
# pip install tiktoken transformers

import json
import statistics
from collections import defaultdict

import tiktoken
from transformers import AutoTokenizer

# 1. A sample of REAL requests, one JSON object per line:
#      {"lang": "hi", "text": "..."}
#    Aim for a few hundred per language, drawn from production
#    traffic rather than from a translated benchmark set.
SAMPLE_PATH = "traffic_sample.jsonl"

# 2. The tokenisers you are actually billed against.
#    Add the ones relevant to you; remove the ones that are not.
TOKENISERS = {
    "cl100k_base": tiktoken.get_encoding("cl100k_base"),
    "llama-3.1":   AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B"),
}

BASELINE_LANG = "en"


def count_tokens(tok, text):
    if hasattr(tok, "encode_ordinary"):          # tiktoken
        return len(tok.encode_ordinary(text))
    return len(tok.encode(text, add_special_tokens=False))


def word_count(text):
    # Whitespace splitting is a rough proxy. It under-counts for
    # agglutinative morphology and for scripts that delimit words
    # differently from Latin, which is why tokens-per-character is
    # reported alongside it. Treat both as estimates, not truth.
    return max(1, len(text.split()))


samples = defaultdict(list)

with open(SAMPLE_PATH, encoding="utf-8") as fh:
    for line in fh:
        rec = json.loads(line)
        text, lang = rec["text"], rec["lang"]
        words = word_count(text)
        chars = max(1, len(text))
        for name, tok in TOKENISERS.items():
            n = count_tokens(tok, text)
            samples[(name, lang)].append((n / words, n / chars))

# Median, not mean: one pathological request should not set your budget.
fertility = {
    key: (
        statistics.median(p[0] for p in pairs),
        statistics.median(p[1] for p in pairs),
        len(pairs),
    )
    for key, pairs in samples.items()
}

for (name, lang), (tpw, tpc, n) in sorted(fertility.items()):
    base = fertility.get((name, BASELINE_LANG))
    tax = tpw / base[0] if base else float("nan")
    print(name, lang, "tok/word", round(tpw, 2),
          "tok/char", round(tpc, 3),
          "tax", round(tax, 2), "n", n)

No sample output is printed here, and that is deliberate. The numbers this produces depend entirely on your corpus and your tokeniser list, and any illustrative output would be invented. Run it on your own data; the result is the only version of this figure that governs your invoice.

The second step converts fertility into money. This is the conversation that gets a budget approved, and it needs to be in the currency your finance team reports in.

# cost_from_fertility.py — turn measured fertility into money.
#
# There is deliberately no rate card in this file. Provider prices
# move, and a stale hard-coded rate is worse than no rate. Fill in
# your own contracted numbers and your own measured fertility.

RATE_PER_M_TOKENS = None      # your input price per 1M tokens, in USD
USD_TO_INR        = None      # your finance team's rate for the period
USD_TO_GBP        = None

# From fertility.py: MEDIAN tokens per word, per language, on your corpus.
FERTILITY = {
    "en": None,
    "hi": None,
    "ta": None,
    "ml": None,
}

AVG_WORDS_PER_REQUEST = None  # measure this from the same sample


def tokens_per_1k_requests(lang):
    return FERTILITY[lang] * AVG_WORDS_PER_REQUEST * 1_000


def cost_per_1k_requests(lang):
    tokens = tokens_per_1k_requests(lang)
    usd = tokens / 1_000_000 * RATE_PER_M_TOKENS
    return {
        "tokens": tokens,
        "usd": usd,
        "inr": usd * USD_TO_INR,
        "gbp": usd * USD_TO_GBP,
    }


def tax_vs_english(lang):
    # The multiplier survives every price change. The currency
    # figure does not. Report both, and lead with the multiplier
    # when you are arguing for engineering time.
    return FERTILITY[lang] / FERTILITY["en"]


def annual_delta(lang, requests_per_month):
    """What you would save if this language cost what English costs."""
    per_1k = cost_per_1k_requests(lang)["usd"]
    english = cost_per_1k_requests("en")["usd"]
    return (per_1k - english) * (requests_per_month / 1_000) * 12

That last function is the one to take into the prioritisation meeting. A fertility multiplier is an interesting fact; an annual figure in rupees or pounds for the gap between what a language costs and what it would cost at English fertility is a decision.

The mitigation ladder, cheapest first

Work down this list in order. Each rung costs more than the one above it, and the cheap rungs frequently make the expensive ones unnecessary.

Lever	Typical effort	What it changes	Trade-off
Measure and re-baseline the cost model	Hours to days	Nothing in the product; everything in the forecast and the context budget	None. The cost of skipping it is a cost model that is wrong by an unknown multiple
Prompt hygiene natively in the target language	Days	Removes duplicated English scaffolding and translate-then-answer round trips	Needs native-speaker review; more prompt variants to maintain and evaluate
Choose a model whose tokeniser covers the script	Days to weeks, mostly evaluation	Fertility drops to that tokeniser's level for the whole workload	A whole-model swap: quality, latency, licence, availability and evals all change
Vocabulary extension and continued pre-training	Weeks to months, plus GPU budget	Extends the merge table to cover your scripts on a model you already like	You now own and must maintain a model, including re-evaluation and safety testing
Translation-pivot architecture	Weeks	Runs the expensive reasoning stage on English text	Quality and cultural-fidelity loss, extra latency, two additional failure modes
Distil a small task-specific model with a suitable tokeniser	Months	Lowest steady-state cost per request for one narrow task	Narrow scope only; needs training data, evaluation and permanent ownership

Rung one and two: measure, then write natively

The first rung is free and has already been described. The second is nearly free and is skipped surprisingly often. Many multilingual products are built as English products with a translation step bolted on: the system prompt is English, the instructions are English, the examples are English, and the model is asked to answer in Hindi. That design pays the fertility tax on the output and often on an intermediate translation as well, while also producing the stilted register that native speakers immediately recognise. Writing the system prompt and examples in the target language, and asking for the answer directly, removes a round trip and usually improves the output. Our guide to prompting across Indic languages and British English in one application covers the mechanics of maintaining several prompt sets without them drifting apart.

Rung three: pick a tokeniser that knows your script

This is the highest-leverage single move available, because it addresses the cause rather than the symptom. Tokenisers built with Indic scripts in scope report dramatically different fertility. Sarvam-1's tokeniser achieves near-English token fertility of roughly 1.4 to 2.1 tokens per word for Indic scripts. Set that against 12.39 tokens per word for Tamil under the Llama-3.1 tokeniser and the scale of what a tokeniser choice determines becomes obvious — this is not a few percentage points, it is most of the bill.

The honest caveat is that you cannot swap a tokeniser without swapping the model, and swapping the model changes your task quality, your latency profile, your licence position, your regional availability and every evaluation baseline you have. Treat it as an evaluation project with a cost hypothesis attached, not as a procurement decision. Build the eval on your own tasks in the target language first, or you will be comparing an English-tuned baseline against a fair test.

Rung four: vocabulary extension and continued pre-training

If you are committed to a particular model family, you can extend its vocabulary with tokens covering your scripts and continue pre-training so the model learns to use them. This is a real technique with published results: IndicSuperTokenizer reports improving average fertility score by 39.5 per cent over LLaMA and 18 per cent over Sutra.

It is also genuinely expensive, and the trade-off deserves stating plainly. Extending a vocabulary means new embedding rows that start untrained, so you need enough continued pre-training for the model to use them competently — and that training can move behaviour on tasks you were not trying to change. You end the exercise owning a model: your own evaluation suite, your own regression testing, your own safety assessment, your own deployment. For a team whose entire product is one language pair at large volume, that arithmetic can work comfortably. For most teams it does not, and rung three gets most of the benefit for a fraction of the commitment.

Rung five: translation pivots, covered honestly

Translate the user's input into English, do the expensive reasoning in English where fertility is lowest, translate the answer back. It is a legitimate architecture and it does reduce token spend, particularly where the reasoning stage is long and the user-facing text is short.

The costs are equally real. Every translation step is a lossy compression of meaning, and what it loses first is exactly what matters most in customer-facing work: register, honorifics, idiom, culturally specific reference, and the terms of art of a domain that may not have settled English equivalents. You have added two model calls to the latency budget and two new failure modes, and when the output is wrong it is now materially harder to work out which stage caused it. Legal, medical and financial content is where this bites hardest, and it is precisely where teams reach for the pattern because the reasoning is expensive.

Rung six: distil something small and specific

At the bottom of the ladder, for a narrow high-volume task, you can train a small task-specific model on a tokeniser appropriate to the script. Steady-state cost per request is the lowest of any option here, and for a single well-defined task the quality can be entirely adequate. The commitment is the largest: training data, an evaluation harness, and permanent ownership of a model that will need retraining as the task drifts. Our guide to teacher-student distillation in production covers when that trade is worth making, and the same reasoning applies to embedding models — if your retrieval stack is multilingual, the tokeniser question applies there too, which is one of the things a proper retrieval benchmark of your own will surface.

When not to fix it

The most expensive mistake available here is not ignoring the tokeniser tax. It is spending three engineer-months on it when the annual saving is smaller than the salary cost of the work.

The break-even reasoning is straightforward and worth doing explicitly before anyone opens a branch. Take the annual delta from the measurement script — the difference between what your non-English traffic costs and what it would cost at English fertility. Multiply by the share of that delta a given rung can realistically recover, which is well under 100 per cent for every rung. Compare against the fully loaded cost of the engineering, plus the ongoing maintenance the rung creates. Rungs four and six create permanent maintenance obligations; count them.

Three situations where the answer is usually to leave it alone. If non-English traffic is a small share of volume, the multiplier applies to a small base and the absolute saving stays small even when the ratio looks alarming — a 13 times multiplier on two per cent of traffic is a smaller line than a 20 per cent inefficiency on the other 98. If the workload is short-form — classification, routing, extraction, anything where inputs are a sentence and outputs are a label — the token counts are small enough that a multiplier on them rarely justifies a model migration. And if the language mix is still changing because the product is early, measure and wait, because optimising a traffic distribution that will look different in two quarters is wasted work.

Not an India-only problem

Indic scripts are the sharpest case in the published literature, but the mechanism has nothing India-specific about it. It is a property of merge tables trained on English-centric corpora, and it applies to any language under-represented in that training.

A related paper measured the same effect across 24 to 25 European languages (arXiv:2605.24718), which settles the question of whether this is a South Asian curiosity. It is not. The pattern is the general one: the further a language sits from the training distribution — by script, by morphology, by sheer corpus volume — the more the merge table fails on it and the higher the fertility.

For a UK team, three practical consequences follow, and none of them require serving Indian languages at all.

Your per-user cost varies by language, and your pricing probably does not. A flat per-seat price across a multilingual European customer base means some cohorts are materially more expensive to serve than others. That is fine if you know it and have decided to absorb it; it is a problem if the mix shifts and nobody was watching. This is the same left-tail problem that shows up in any margin-per-customer analysis, arriving through a different door.

Your evaluation suite is probably monolingual, and it flatters you. If your context-window behaviour, truncation logic and latency budgets were all validated on English, they have not been tested under the conditions your non-English users actually experience. Add at least one high-fertility language to the regression suite specifically to exercise the truncation and cache paths.

Data-residency and model-choice constraints interact with this. UK and EU teams frequently have a shortlist of permissible models for regulatory or contractual reasons, and rung three — swapping to a model with a better tokeniser — may simply not be available within that shortlist. If so, the ladder for you effectively starts at rung two and jumps to rung four or six, and the break-even maths changes accordingly. Work that out before you promise a saving.

What to re-check every six months

Everything above is stable in mechanism and unstable in numbers, which is the defining property of a fact worth putting on a calendar rather than in your memory.

Model families change tokenisers between major versions. A vendor that ships a new tokeniser with a broader vocabulary can move your fertility substantially in either direction without any announcement that reads as relevant to cost, because the release notes will discuss quality. That means every model upgrade is a reason to re-run the measurement, before the upgrade if you can and immediately after if you cannot. Treat a tokeniser change like a price change, because that is functionally what it is.

Set a recurring review with four items on it. Re-run the fertility script against your current traffic mix and your current tokeniser list, since both drift. Re-derive the annual delta per language, because a rung that failed the break-even test last time may pass it now. Re-check whether new models with better-covered tokenisers have entered your permissible shortlist, and whether vendor efficiency claims made since your last review have acquired independent verification. And re-examine your language mix, because in a growing product it moves faster than almost anything else in the cost model.

As of September 2026 the figures in this article are the best public ones available. They will age. The measurement method will not, and neither will the reason it matters: a token is not a neutral unit of work, and any cost model, context budget or evaluation suite that assumes it is will be wrong for every user who does not write in English. More cost engineering guides are collected in our tips section.