Speculative decoding: faster LLM inference without quality loss

What you need to know

• It is lossless. Speculative decoding accelerates generation by drafting several tokens cheaply and verifying them in one target-model pass. A rejection-sampling step guarantees the output distribution is identical to plain decoding — you are not trading quality for speed.
• The speedup comes from acceptance rate. The more of the draft the target accepts, the fewer expensive forward passes you run. The original method reported roughly 2-3x; modern feature-level drafters report more on favourable workloads.
• Batch size is the deciding variable. Gains are largest at low batch / latency-bound serving and shrink as large batches make you compute-bound. This is the single most misunderstood point.
• You have four families to choose from. Separate draft model, Medusa heads, EAGLE feature-level drafting, and cheap n-gram lookup — each with a different training and memory cost.
• It is well supported. As of July 2026, vLLM, TensorRT-LLM, SGLang, llama.cpp and Hugging Face TGI all ship some form of speculative decoding, usually behind one config block.

How speculative decoding works — and why it is lossless

Autoregressive generation is slow for a structural reason: each token depends on the one before it, so a large model must run a full forward pass per token. At batch size one that pass is dominated by reading the model weights from GPU memory, not by arithmetic. The GPU spends most of its time waiting on memory bandwidth while its compute units sit idle. Speculative decoding turns that idle compute into free tokens.

The loop has three steps. First, draft: a cheap mechanism proposes the next k tokens — say four or five — far faster than the big model could. Second, verify: the large target model runs a single forward pass over all k drafted tokens at once, scoring each in parallel. Third, accept or reject: a rejection-sampling rule walks the draft left to right, accepting each token with a probability derived from the ratio of the target and draft distributions, and stopping at the first rejection. On rejection it resamples one corrected token from an adjusted distribution.

The elegant part is the guarantee. Leviathan et al. showed that this accept/reject procedure — sometimes called speculative sampling — produces tokens distributed exactly as the target model would produce on its own, up to floating-point tie-breaking. It is not an approximation, not a distillation, not a quality/speed dial. The output is identical; only the number of expensive target passes changes. Their paper reported a 2X-3X acceleration on T5X with no change to outputs, and DeepMind's Chen et al. published the same core result independently the following year (arXiv 2211.17192; arXiv 2302.01318; see also Google Research's retrospective).

Because a rejection always costs you one target pass anyway, the whole thing is a bet: if the draft is good, you cash several tokens per verification pass; if it is bad, you fall back to roughly normal speed. That is why it can never make you slower by more than the small overhead of running the drafter — and why the acceptance rate is the number that governs everything. A drafter that the target accepts 80% of the time buys a large speedup; one accepted 30% of the time barely helps.

The techniques, compared

"Speculative decoding" is an umbrella. The families differ in where the draft comes from, and that single choice drives your training effort, memory cost and speedup. Here is how the four main approaches line up.

Technique	Separate model?	Extra training?	Memory cost	Typical speedup	Best-fit workload
Draft model (two-model)	Yes — a small model of the same family	None if a small sibling exists	High — a second model in VRAM	~2-3x, workload-dependent	You already have a small + large model with a shared tokeniser
Medusa (extra heads)	No — heads bolted onto the target	Yes — fine-tune the heads	Low — a few small heads	~2.2-3.6x reported	No suitable draft model exists; you can fine-tune
EAGLE / EAGLE-2 / EAGLE-3	No — a lightweight feature-level drafter	Yes — train the EAGLE head	Low-to-moderate	State-of-the-art; EAGLE-3 reports up to ~6.5x on favourable runs	Latency-critical single-stream serving where you can train a head
N-gram / prompt lookahead	No — no model at all	None	Negligible	Modest, but effectively free	Repetitive output: summarisation, code edits, RAG that quotes context

Draft-model (two-model) speculative decoding

The original formulation: pair a small draft model with a large target from the same family — for example a 1B drafter fronting a 70B target — so the two share a vocabulary. The draft model autoregressively proposes tokens; the target verifies them in one pass. It needs no new training if a suitable small sibling already exists, but it is the most memory-hungry option because you are holding a second model in VRAM, and it is fiddly to tune the draft length.

Self-speculative and model-internal methods

These attach the drafting apparatus to the target model itself, avoiding a second model entirely.

• Medusa adds several extra decoding heads on top of a frozen (or jointly tuned) backbone, each predicting a token a few positions ahead, then verifies candidate continuations with tree attention. It reported 2.2-3.6x speedups without degrading quality and is simple to bolt on (arXiv 2401.10774; reference implementation).
• EAGLE performs autoregression at the feature level — it drafts using the target model's own hidden states rather than raw tokens, which makes the draft far more accurate. EAGLE-2 added a dynamic draft tree; EAGLE-3 dropped feature prediction for direct token prediction with multi-layer fusion and reported up to ~6.5x on favourable workloads, roughly 1.4x over EAGLE-2 (EAGLE, arXiv 2401.15077; EAGLE-3, arXiv 2503.01840). Treat the headline multiples as best-case; real serving sees less.
• Lookahead decoding takes a different tack again, using a parallel Jacobi-style iteration to generate and verify n-grams without any auxiliary model or training. It is a useful option when you cannot train anything.

N-gram and prompt lookahead — no draft model at all

The cheapest family drafts by copying. N-gram (or prompt-lookup) speculation scans the prompt and generated text so far for a matching suffix and proposes the tokens that followed it last time. There is no model, no training and essentially no VRAM cost. It shines precisely where a learned drafter struggles: highly repetitive output such as retrieval-augmented answers that quote their context, code edits that echo the input, or structured extraction. It does little on free-form creative text, but it is close to free to try.

The trade-offs that decide your speedup

The technique table tells you what to pick; these four trade-offs tell you whether you will actually see the numbers.

Draft quality versus draft cost

Every drafted token costs something. A bigger, more accurate drafter raises the acceptance rate but eats into the time it saves; too cheap a drafter is accepted so rarely that you pay for drafts you throw away. This is why EAGLE's feature-level approach wins — it buys high acceptance from a very light head. When you tune, watch the product of acceptance rate and draft length, not either one alone.

Batch size: the point everyone gets wrong

This is the trade-off that overturns naive expectations. Speculative decoding converts spare compute into saved memory-bandwidth-bound steps. At batch size one, the GPU has abundant idle compute, so speculation is nearly free and gains are large. As batch size climbs, the server processes many sequences per weight read and becomes compute-bound — the spare capacity that speculation was spending simply is not there any more. Independent benchmarking bears this out: gains shrink markedly with batch size, with one study reporting EAGLE dropping from roughly 1.96x at batch size 1 toward ~1.21x at batch size 128 (Batch Speculative Decoding Done Right, arXiv 2510.22876; SpecDecode-Bench).

GPU memory for the drafter or heads

A separate draft model is the expensive choice — a second set of weights and its own KV cache compete for VRAM you would otherwise spend on batch size or context length. Medusa heads and EAGLE drafters are far lighter. On memory-constrained cards — the kind many builders rent through IndiaAI compute or piece together from consumer GPUs in a UK home lab — this can decide the whole approach: the two-model route may simply not fit alongside a usable batch.

Tokeniser and vocabulary alignment

For classic two-model speculative decoding, the draft and target must share a tokeniser and vocabulary, because the accept/reject test compares their per-token probabilities directly. Mismatched vocabularies break the guarantee; cross-vocabulary methods exist but add real complexity. Self-speculative methods dodge this entirely — Medusa and EAGLE heads live inside the target model, so there is only ever one vocabulary to reason about. If you cannot find a small sibling with the same tokeniser as your target, that alone is a strong reason to prefer EAGLE or Medusa.

Where it is supported, and how to switch it on

As of July 2026, speculative decoding is a first-class feature in every major serving stack, usually a single config block rather than a code change.

• vLLM exposes draft-model, EAGLE and n-gram methods through a --speculative-config object. The EAGLE line has been co-developed with the EAGLE authors, with EAGLE 3.1 landing in 2026 (vLLM docs; EAGLE 3.1 announcement).
• TensorRT-LLM builds speculative modes into the engine at compile time and ships EAGLE-3 heads for several popular models; it is the route for teams already committed to NVIDIA's Triton stack.
• SGLang has a strong EAGLE implementation with configurable tree attention and, per its own reporting, often leads on EAGLE specifically; it supports EAGLE-3 checkpoints via --speculative-algorithm.
• llama.cpp supports draft-model speculation and prompt/n-gram lookup, which is ideal for local and edge deployments; EAGLE-3 was not yet supported at the time of writing.
• Hugging Face TGI supports Medusa and n-gram-style speculation for teams standardised on the TGI serving path.

Enabling n-gram speculation in vLLM is about as low-risk as an optimisation gets — no extra weights, no training:

# vLLM — n-gram (prompt-lookup) speculation. Zero extra VRAM.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

# vLLM — EAGLE draft head for higher acceptance on free-form text.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{
    "method": "eagle",
    "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-70B",
    "num_speculative_tokens": 5
  }'

The output is identical with or without these flags — that is the whole point. So the only questions are whether the speedup materialises at your batch size, and whether the drafter fits in memory. Both are measurable in an afternoon.

If you are tuning the broader serving picture, speculative decoding slots in beside the other levers we cover in the vLLM throughput and latency playbook and the cost, quantisation and batching guide.

A decision rule for builders self-hosting

Whether you are drawing on IndiaAI GPU allocations, a UK or EU on-premise cluster kept in-region for data-residency reasons, or rented capacity you are paying for by the hour, the same rule applies. Speculative decoding is a latency optimisation that spends compute to save wall-clock time. Reach for it when:

• Latency is the product. Interactive assistants, coding copilots, voice agents — anywhere a human waits on the last token and your batch sizes are small. This is the sweet spot.
• You have compute headroom. Your GPUs are underutilised at your real load, so there is idle compute for speculation to spend.
• Your output is repetitive. Start with n-gram lookup: it is free to try and often wins outright on RAG, summarisation and code editing.

Reach for batching and quantisation instead when:

• You are optimising cost-per-token at high concurrency. Large-batch, compute-bound serving is where speculation fades and continuous batching shines.
• VRAM is your binding constraint. A separate draft model may be exactly the memory you cannot spare — quantising the target frees more headroom per rupee or pound.

The pragmatic answer for most teams is to stack them: quantise and batch for throughput and cost, then add EAGLE or n-gram speculation on the latency-sensitive routes. Speculative decoding is not a substitute for good batching and caching — it is a complement. For the cost side of that stack, our guide to caching, routing and compression covers the levers that compound with it.

The bottom line

Speculative decoding is one of the few genuinely free lunches in LLM serving: measurably faster generation with a mathematical guarantee that the output does not change. The catch is not quality — it is that the speedup is contingent. It scales with acceptance rate and evaporates as batch size pushes you compute-bound. Pick the technique that matches what you can train and what fits in memory, benchmark honestly at production load, and treat it as a latency lever that sits alongside batching, quantisation and caching rather than replacing any of them. Get that framing right and it is one of the highest-leverage changes you can make to a self-hosted stack.

One last framing for teams weighing the effort. The builders who get the most out of speculative decoding tend to be the ones serving latency-sensitive, low-concurrency traffic — a coding assistant, an internal copilot, a single-tenant agent — on their own GPUs, whether that is a rented A100 in an Indian data centre or a workstation card under a desk in Manchester. If your workload is high-throughput batch inference, your gains will be smaller and quantisation or better batching will usually repay the effort faster. As of July 2026 the safest way to decide is empirical: enable a draft model or an EAGLE head behind a feature flag, measure tokens-per-second and acceptance rate at your real batch size, and keep it only if the numbers hold at production load rather than at batch size one.

Primary sources worth reading in full: Leviathan et al. (2022/2023), Chen et al. (2023), Medusa, EAGLE-3, and the vLLM speculative decoding docs.