Prompt Compression in Production: Cutting 2-5x Tokens Without Losing the Answer

What compression actually buys you

Input tokens are charged on every call and prefilled on every call. The second half is the one teams under-weight. The bill turns up in a monthly invoice and someone eventually notices it; the latency is invisible in aggregate and extremely visible to the person watching a spinner. On a request with a long context and a short answer — most retrieval-augmented question answering, most classification, most support triage — the dominant contributor to time-to-first-token is prefilling your input. Halve the input and you take a real bite out of perceived responsiveness as well as cost.

Compression sits alongside two other levers, and the three are genuinely orthogonal. Caching helps when the same prefix is seen again: it does nothing at all for a call whose context is unique. Routing helps when a query is easy enough for a cheaper model: it does nothing for a query that genuinely needs your best model. Compression helps for the long single call — the one where the context differs every time, the query is hard, and there is simply a great deal of text going in. If you have not yet built the other two, build them first; we cover the whole picture in the guide to cache, route and compress as a cost strategy, and this article deliberately goes deeper on the compression pillar alone rather than repeating that material.

Be honest about the ceiling before you start. Compression is a quality-for-cost trade, not free money. Every token you remove is a token the model cannot attend to, and somewhere on the ratio curve is the point where the next token removed is the fact the answer depended on. The discipline is not "how much can I cut" but "where is the knee, and am I still safely behind it". Teams that skip that question ship a 5x compression, watch the bill fall, and discover months later that multi-hop queries have been quietly wrong ever since.

Compression is worth its complexity in proportion to how long and how variable your contexts are. A 900-token prompt is not worth a stage, because the compressor's own latency swamps the saving; an 8,000-token prompt with six retrieved documents is. A team running a 200,000-token context on every call should first ask whether that context should exist at all, which is the subject of the long context versus retrieval decision.

Four families of compression

The word "compression" covers at least four quite different mechanisms with different ratios, different risks and different infrastructure costs. Confusing them is the most common reason teams either over-engineer or get burned.

Manual and structural. Rewriting the system prompt that grew by accretion over eighteen months. Converting three paragraphs of prose instruction into a seven-item list. Deleting the four few-shot examples that were added to fix a failure mode that a model upgrade fixed anyway. Replacing a verbose natural-language schema description with a compact one. This family typically yields somewhere in the region of twenty to forty per cent reduction on a prompt that has never been pruned, at effectively zero quality cost, with no tooling, no new dependency and no latency on the request path. Do this first, always. It is unglamorous and there is no paper about it, and it is routinely the single largest win available. It also makes everything downstream easier, because a tidy prompt is one whose spans you can actually classify as compressible or protected.

Token-level pruning. The LLMLingua family, from Microsoft Research, uses a small language model to score each token by how surprising it is in context and drops the least informative ones — natural language is redundant, and a large model can reconstruct meaning from a sparser signal than a human reader needs. The original LLMLingua paper reports up to 20x compression with little performance loss on its evaluation set; LongLLMLingua extends the approach to long-context and multi-document settings. The practical objection to the first generation was speed, since computing perplexity over your whole context is itself an inference pass. LLMLingua-2 answers that by reframing compression as token classification with a bidirectional encoder, trained by data distillation on GPT-4's compression decisions, and reports running several times faster than its perplexity-based predecessors while generalising better across domains. The reference implementation is the microsoft/LLMLingua repository.

Sentence-level selection. Selective Context, from the work of Li and colleagues on compressing context to enhance inference efficiency, scores units of text by self-information and drops those below a threshold. The important engineering difference from token-level pruning is that it operates at sentence or phrase granularity rather than at token granularity, so sentence boundaries survive. That matters more than it sounds: structured or semi-structured content that would be shredded by token pruning often passes through sentence selection intact, because either a whole sentence goes or none of it does. If your context is a mixture of prose and lightly structured material, this is a gentler starting point.

Extractive re-ranking. For retrieval-augmented generation specifically, the highest-value move is usually not to compress text at all but to include less of it. Run a cross-encoder reranker over the passages your retriever returned and keep only the top-k. You get a large token reduction, every surviving passage is byte-for-byte intact so quotations and citations still work, and you are adding a component that improves retrieval precision on its own merits. For most multi-document question answering this beats token pruning on every axis that matters. The mechanics are covered in the guide to reranking with cross-encoders, ColBERT and hosted rerankers.

A fifth option is worth naming so you can decline it deliberately: abstractive compression, where a model summarises the context before it reaches the answering model. The RECOMP work studies both extractive and abstractive compressors for retrieval-augmented models and is worth reading. The production risk is specific: an abstractive compressor is a generative model, and generative models hallucinate. A hallucination introduced during compression enters your context as though it were retrieved fact, and the answering model cannot tell the difference. Use it on content the user can verify, never on content that grounds a factual claim.

Where compression is safe and where it is not

The single most useful thing you can do before wiring up any compressor is to classify the spans of your prompt. Not all context is the same kind of text, and a technique that is nearly free on one kind is destructive on another. Do this as an explicit exercise, write the classification down, and encode it in your prompt builder so that a new engineer adding a new span has to declare which bucket it falls into.

The mechanism behind the hard "no" rows is worth understanding, because it is what stops people making an exception. Token-level pruning drops the tokens a scoring model finds least surprising. In prose, those are function words and filler, and losing them is largely harmless. In structured content, the unsurprising tokens are the structure itself: closing braces, the commas between fields, the SELECT and FROM keywords, quotation marks, indentation. They are low-entropy precisely because they are mandatory, so a compressor optimising for information content removes them first. What comes out looks like code, looks like JSON, and is wrong in ways downstream validation will not reliably catch. If your outputs must be parseable, keep the schema pristine — see the guide to structured output patterns in production.

Wiring it in: a compression stage in the request path

The right shape for a compression stage is tiered: cheapest and safest techniques first, learned compressors last, and a guard at the top that skips the whole thing when the prompt is too short to be worth it. That guard matters more than it looks. A compression stage has its own latency and its own model cost, and on a 1,200-token prompt you can easily spend more time and money compressing than you save. Below the threshold, the correct action is to do nothing at all.

"""Tiered prompt compression in the request path.

Order matters: cheap deterministic wins first, learned compressor last.
The whole stage is skipped when the prompt is small enough that the
compressor's own latency would dominate the saving.

count_tokens / render / reranker / summarise_turns are your own helpers.
"""

from dataclasses import dataclass, field

MIN_TOKENS_TO_BOTHER = 2000    # below this the stage is pure overhead
DEFAULT_TARGET_RATIO = 2.5     # measured against a golden set, not guessed
KEEP_VERBATIM_TURNS  = 6       # recent turns are never touched
ENABLE_TOKEN_PRUNING = False   # off until the sweep says otherwise


@dataclass
class CompressionResult:
    prompt: str
    original_tokens: int
    final_tokens: int
    stages: list = field(default_factory=list)

    @property
    def ratio(self) -> float:
        return self.original_tokens / max(self.final_tokens, 1)


def build_prompt(
    system_block,      # protected: guardrails and safety instructions live here
    retrieved,         # list of (doc_id, text) from the retriever
    history,           # prior turns, oldest first
    user_turn,         # protected: never compressed
    query,
    target_ratio=DEFAULT_TARGET_RATIO,
) -> CompressionResult:

    original = (
        count_tokens(system_block)
        + sum(count_tokens(t) for _, t in retrieved)
        + sum(count_tokens(t) for t in history)
        + count_tokens(user_turn)
    )

    if original < MIN_TOKENS_TO_BOTHER:
        body = render(system_block, retrieved, history, user_turn)
        return CompressionResult(body, original, original, ["skip:short-prompt"])

    stages = []

    # Stage 1 - extractive. Rerank retrieved passages, keep the top-k only.
    # Best value per millisecond for anything RAG-shaped, and every
    # surviving passage stays byte-for-byte intact.
    if retrieved:
        keep = max(3, int(len(retrieved) / target_ratio))
        ranked = reranker.rank(query, [t for _, t in retrieved])
        retrieved = [retrieved[i] for i in ranked.top_indices(keep)]
        stages.append(f"rerank:kept-{keep}-of-{len(ranked)}")

    # Stage 2 - structured history. Roll up old turns into a summary,
    # keep the most recent turns verbatim so exact wording survives.
    if len(history) > KEEP_VERBATIM_TURNS:
        old, recent = history[:-KEEP_VERBATIM_TURNS], history[-KEEP_VERBATIM_TURNS:]
        history = [summarise_turns(old)] + recent
        stages.append(f"history:rolled-up-{len(old)}-turns")

    body = render(system_block, retrieved, history, user_turn)
    final = count_tokens(body)
    budget = int(original / target_ratio)

    # Stage 3 - learned compressor, prose spans only, and only if the
    # first two stages left us short of budget. Protected spans - the
    # system block, code, JSON, tables - are never passed in.
    if ENABLE_TOKEN_PRUNING and final > budget:
        prose_spans = extract_prose_spans(body)
        rate = max(0.25, budget / final)
        body = splice(body, compress_prose(prose_spans, rate=rate))
        final = count_tokens(body)
        stages.append(f"prune:rate-{rate:.2f}")

    return CompressionResult(body, original, final, stages)

Three choices in that sketch are deliberate. The system block never enters a compressor, because it carries your guardrails. The user's current turn never enters one either, because it is short and it is what the answer has to be about. And ENABLE_TOKEN_PRUNING defaults to off, because the first two stages are usually sufficient and the third is the one that puts a model on your critical path. Log stages and both token counts on every request; reconstructing them afterwards is impossible.

If you do reach for the learned compressor, the LLMLingua-2 call shape is small.

# pip install llmlingua
from llmlingua import PromptCompressor

# Loaded once at process start, not per request. This is a several-hundred-
# million-parameter encoder: it wants a GPU if it is going to sit on the
# critical path of a live endpoint.
compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
    device_map="cuda",
)

def compress_prose(chunks, rate=0.4):
    """chunks: list of prose strings. rate is the fraction of tokens KEPT,
    so rate=0.4 targets roughly 2.5x compression."""
    result = compressor.compress_prompt(
        chunks,
        rate=rate,
        # Protect the punctuation that carries structure. Without this,
        # newlines and separators are among the first things pruned.
        force_tokens=["\n", ".", "?", ",", ":"],
        force_reserve_digit=True,   # keep digits in figures, dates, IDs
        drop_consecutive=True,
    )
    # result also carries origin_tokens, compressed_tokens and rate -
    # emit all three as metrics on every call.
    return result["compressed_prompt"]

Be clear-eyed about what you have just added. The compressor is a model that must be loaded, warmed, scaled, monitored and upgraded, and on a latency-sensitive endpoint it wants an accelerator of its own — a real infrastructure line item sitting on the path of every request. That is why the extractive path wins so often for live traffic: a hosted reranker call or an in-process template pass needs no GPU of yours. Save the learned compressor for batch and asynchronous work, where an extra hundred milliseconds is free and the batch APIs are already cutting your rate, or for endpoints whose contexts are long enough that the arithmetic is not close.

Choosing a ratio: the compression-quality curve

Here is the discipline that separates a compression stage you can defend from one you will quietly roll back. You do not pick a ratio. You measure one. The ratio that suits a triage classifier over short tickets is not the ratio that suits a legal-document question answering system, and neither is the ratio in a paper's headline. Published numbers are real results on the datasets the authors evaluated; they are not a prediction about your traffic.

The procedure takes an afternoon once you have a golden set. Sweep the ratio across a range — say 1x, 1.5x, 2x, 3x, 5x, 8x, 12x — running your task metric at each point, and plot accuracy against ratio. The shape you almost always get is a plateau, then a knee, then a cliff; sit comfortably back from the knee rather than balanced on it, because traffic drifts and prompts change. Then repeat the sweep per traffic segment, because the aggregate curve hides the fact that different query types have knees in different places.

Treat every figure in that table as a starting point for your first sweep, not a guarantee. The bands reflect what tends to hold for prose-heavy English contexts on general tasks; a domain with dense terminology, a language other than English, or a task needing precise recall of specific values will have a knee much earlier. Conversely, a genuinely bloated context — the kind that accumulated over two years of nobody deleting anything — often takes far more compression than these bands suggest, because so much of it was never doing any work.

The eval you need before you ship this

Compression is one of very few optimisations that can degrade quality without producing a single error, exception or alert. Nothing goes red. The system returns confident, well-formed answers that are subtly less correct than they used to be. That makes an eval a precondition rather than a nice-to-have — and if you are not prepared to build one, stop at the manual pass and the reranker and go no further.

The golden set should be 100 to 300 real production prompts with known-good outputs. Real, not synthetic: the point is to capture the messy, ambiguous, badly-punctuated shape of your actual traffic. Sample across segments deliberately rather than taking the most recent 200, and over-represent the hard cases relative to their traffic share, because that is where compression fails and exactly what a random sample under-weights.

Pick a task metric that matches what the endpoint is for. Classification and extraction get exact match or F1. Retrieval-augmented answering gets retrieval recall at the compressed context — did the passage containing the answer survive? — alongside answer quality. Open-ended generation gets a rubric-scored judge, with all the calibration care that implies; the guide to LLM-as-a-judge rubrics, bias and calibration covers why an uncalibrated judge will happily tell you nothing changed. Run the metric at every point on the sweep and store per-example results, not just aggregates.

Watch the tail, not the mean. Compression failures are not evenly distributed — they concentrate on the examples that were already hardest, the ones needing information from several places at once, the ones where the answer hinged on a qualifier. A mean that moves from 0.87 to 0.86 looks like noise and frequently hides a bottom decile that moved from 0.71 to 0.44. Report the aggregate alongside the metric on your hardest quartile, and gate on the second one.

Then put the whole thing in continuous integration, because a ratio validated once is a ratio that will drift. Wire the sweep into the harness that already guards your prompts, as described in the guide to running evals in CI for prompt and agent regression testing, and let any change to a prompt template, retriever configuration or compression parameter trigger a re-run.

A worked cost and latency model

Naive compression arithmetic says: cut the input tokens in half, halve the input bill. The real number is smaller, for two reasons that are both easy to forget. Only part of your prompt is compressible, and the compressor has costs of its own. Here is a worked example with every assumption stated so you can substitute your own.

Take a support-triage endpoint — the same shape whether it belongs to a Bengaluru fintech or a Manchester health-tech. Assumptions: 40,000 requests per day; 8,000 input tokens per request, made up of a 1,200-token protected block (system prompt, guardrails, the user's current message) and a 6,800-token compressible span (six retrieved documents plus older chat history); 300 output tokens. Illustrative model pricing of $3.00 per million input tokens and $15.00 per million output tokens — a mid-tier rate as of August 2026, which you must re-check against the vendor's own pricing page, because these numbers change several times a year. Compressor infrastructure: two accelerator instances at roughly $1.00 per hour to cover peak rather than mean load, so $48 per day. Latency model: a fixed 200 ms of overhead plus roughly 0.15 ms per input token of prefill, with the compressor adding about 110 ms including its network hop.

The light configuration nets $360 per day rather than the $480 the naive calculation promised — about three-quarters of the headline, or roughly $10,800 a month. That is still an excellent return for a week of engineering, and the 400 ms off time-to-first-token is arguably worth more than the money. But notice what the arithmetic exposes. The protected span sets a floor: no amount of compression takes this endpoint below 1,200 input tokens, so the marginal return falls away sharply as the ratio climbs. And the compressor's fixed daily cost means the exercise only makes sense at volume — at 4,000 requests a day the same $48 charge exceeds the $40.80 of input saving outright, so the stage would cost you money rather than save it. Run this table with your own numbers before you build anything; it occasionally tells you not to bother.

Common pitfalls

Five failure modes account for most of the compression projects that get rolled back. The first is by a wide margin the most common and the most expensive.

That interaction is worth understanding rather than merely avoiding. Caching and compression apply to different spans of the same prompt. Your stable system block and fixed few-shot examples are cache territory: identical every time, so leave them long and let the cache pay for them. Retrieved documents and conversation history are compression territory: they differ on every call and the cache can do nothing with them. Draw the boundary explicitly in your prompt builder and the two compose instead of fighting. The mechanics of prefix caching are in the guide to prompt caching on Claude, GPT and Gemini, and our coverage of prompt caching in production sets out how the vendor implementations differ.

The second pitfall is compressing the system prompt that carries your safety and guardrail instructions. It is tempting, because system prompts are often the longest static block you own. But a compressor optimising for information density has no concept of which instructions are load-bearing under adversarial input, and the sentence it drops as low-information may be the one stopping your agent from obeying instructions embedded in a retrieved document. Mark the safety block as protected and enforce it in code, so no future refactor can quietly route it through a compressor.

The third is compressing on the client. It looks efficient — you save bandwidth as well as tokens — but it destroys your ability to debug. When a user reports a bad answer you need the exact prompt that was sent, and if compression happened on a mobile device six app versions ago, you cannot reconstruct it. Compress on the server, inside the boundary you control and can observe. If you already front your model calls with a gateway, that is the natural home for the stage; see the guide to LLM gateways.

The fourth follows directly: log the pre-compression prompt, or at minimum a content hash plus the full compression parameters, on every request. When an incident review asks whether compression caused a bad answer, the only way to find out is to replay the request with compression disabled. Without the original you cannot replay, and you end up arguing from intuition about a system whose design premise was measurement. Sampling is a fair compromise — keep every prompt for a small share of traffic, and every prompt behind a negative user signal.

The fifth is ratio drift. You swept the curve, picked 2.5x and shipped. Since then the retriever moved from six passages to ten, someone added a tool-definitions block, and the system prompt gained a section. The prompt your compressor sees is no longer the prompt you validated, and a ratio that sat comfortably behind the knee may now be on it. Re-run the sweep on a schedule as well as on change, and treat the compression configuration as a versioned artefact tied to a validated curve rather than a number in a config file nobody owns.

Where to start

In order, and do not skip ahead. First, measure your token mix: instrument the prompt builder to emit per-span token counts on every request and look at where the tokens actually are. Teams are wrong about this more often than not — the block everyone blames is frequently a tenth of the total, and the retriever nobody has examined since launch is two-thirds of it. It takes an hour, and you cannot compress sensibly without it.

Second, do the manual pass. Read the system prompt end to end and delete everything not earning its place. Convert prose to structure. Remove the few-shot examples fixing a problem your current model does not have. This costs nothing, risks almost nothing, and routinely returns twenty to forty per cent on a prompt that has never been pruned.

Third, if you do retrieval, add a cross-encoder reranker and keep fewer passages. For most RAG systems this is the largest remaining win, it preserves every surviving passage exactly so citations and quotations keep working, and it is the least risky change on the list.

Only then, and only if your own version of the cost table justifies the infrastructure, reach for a learned compressor. Build the golden set, sweep the curve, gate on the tail, put the sweep in CI. If you cannot justify the eval work, that is a legitimate answer: stop at step three, bank the saving you already have, and revisit when volume grows.

A closing thought about the work itself. A compression-quality curve for a real production system — the measured plot, the segment breakdown, the honest net cost table with the compressor's own overhead subtracted — is an unusually good portfolio artefact, whether you built it at a Bengaluru fintech or a Manchester health-tech. It shows measurement discipline, an understanding of where techniques break, and a willingness to publish the number that came in smaller than the headline. That is what separates an engineer who has read the papers from one who has shipped the thing. Browse the Builders already doing that across India and the UK, and add your own.