Self-Host Open LLMs with vLLM: A Throughput and Latency Playbook

When self-hosting on vLLM beats an API

Every team that ships a language-model feature eventually faces the same fork in the road: keep paying a managed API by the token, or self-host an open-weight model on your own graphics processing units? There is no universal answer, but there is a clear decision rule. Self-hosting on vLLM wins when three things are true at once. First, your traffic is high and reasonably steady, so a GPU stays busy rather than idling. Second, you need control the API will not give you — a specific open-weight model, a custom fine-tune, tight data residency, or predictable latency you can tune. Third, your volume is large enough that a fixed GPU-hour bill undercuts a per-token bill. When those three line up, owning the serving stack is genuinely cheaper and gives you levers no vendor exposes.

When they do not, a managed API is almost always the right call. If your traffic is low, spiky, or unpredictable, you pay nothing for idle capacity with an API and you never carry an on-call rota for inference. The instinct to self-host to save money frequently backfires for exactly this reason: a GPU you rent by the hour but keep busy only ten per cent of the day is more expensive than tokens you buy on demand. Self-hosting is an economy of scale and control, not a default.

Two numbers govern everything that follows, and you should be able to recite them in your sleep. TTFT — time-to-first-token — is the delay before the first token appears, and it drives perceived responsiveness: it is the pause the user stares at. TPOT — time-per-output-token — is how quickly tokens stream after that first one, and it drives how fast the answer feels once it starts. As the performance-engineering guidance from Databricks puts it, TTFT is what the user waits on and TPOT is what they read at, and continuous batching is indispensable for any shared, online serving workload. Every knob in this playbook is ultimately a way to trade TTFT, TPOT and total throughput against one another.

The reason self-hosting is even viable is the serving engine itself. As of mid-2026, oneuptime's vLLM deployment analysis reports that vLLM with PagedAttention and continuous batching serves roughly 10 to 24 times faster than a naive or standard implementation of the same model on the same hardware. That multiple is what turns a GPU you rent in AWS Mumbai (ap-south-1) or London (eu-west-2) from an expensive science experiment into a genuinely economical serving tier. Here is the shape of what follows: size the hardware and the model, stand up the server, stream to users, budget TTFT against TPOT, optimise, and autoscale on the right signals.

Prerequisites: sizing the GPU, the model and the KV cache

Before you touch a serve command, you need to know whether your model fits. Two things consume GPU memory, and teams routinely forget the second one. The first is the model weights. Their size is roughly the parameter count multiplied by the bytes per parameter: FP16 is two bytes, so an 8-billion-parameter model needs about 16 GB just for weights, a 13-billion model about 26 GB, and a 70-billion model about 140 GB — which is why the large models demand multiple GPUs or aggressive quantisation. Drop to FP8 or INT8 at roughly one byte per parameter and those numbers roughly halve: the 8B fits in about 8 GB, the 70B in about 70 GB, which is the difference between needing one 80 GB card and needing several.

The second consumer is the KV cache, and it is the one that surprises people in production. Every token in every concurrent request stores a key and value tensor for each attention layer, so the cache grows with both the number of simultaneous sequences and how long each one is. A rough mental model: cache size scales as two (key and value) times the number of layers times the key/value head dimension times the bytes per element times the total tokens in flight. At high concurrency and long context, the KV cache can rival or exceed the weights themselves. If you size a card for weights alone and forget the cache, you will hit an out-of-memory error the moment real traffic arrives with long prompts.

In practice this sorts hardware into tiers. A 24 GB card — the class you rent as a g5 or g6 instance in both Mumbai and London — comfortably serves a quantised 7-to-8-billion model with useful headroom for the KV cache, which covers a large share of production chat and retrieval workloads. An 80 GB card (the A100/H100 class, rented as p4d or p5 instances) is what you reach for to serve 70-billion-class models, or to serve a smaller model at very high concurrency where the KV cache is the constraint. Order-of-magnitude, a 24 GB instance runs on the order of a dollar or two per hour on-demand in both regions, and an 80 GB card several times that — so the model you can shrink into the cheaper tier directly shapes your unit economics.

Picking the model is a quality-versus-cost search, and the rule is simple: choose the smallest open-weight model that clears your quality bar on your own evaluation set, not the biggest one you can afford. At the frontier end, open MoE releases such as the NVIDIA Nemotron 3 Ultra 550B open-weight mixture-of-experts show you can now self-host genuinely large models — though the VRAM maths above tells you what that commitment costs. At the small, efficient end, quantisation-aware releases like Gemma 4 with QAT that runs on 1 GB Arm laptops prove how far a compact model now goes, and models trained for low-bit robustness survive aggressive quantisation better. Benchmark two or three candidates on your data before you commit hardware to any of them.

Serving with vLLM: PagedAttention and continuous batching

Standing up the server is the easy part. Install the engine, point it at a model, and you have an OpenAI-compatible endpoint in one command. The interesting engineering is in the two ideas that make it fast, and understanding them tells you which knobs matter.

PagedAttention treats the KV cache the way an operating system treats memory. Instead of reserving one large contiguous block per sequence — which wastes enormous amounts of VRAM to internal fragmentation and forces you to over-provision for the longest possible output — it splits the cache into small fixed-size blocks that can live anywhere in memory, non-contiguously, and maps them with a lookup table. Fragmentation collapses to almost nothing, so you can pack far more concurrent sequences into the same card. Continuous batching is the scheduling counterpart: rather than forming a static batch and making every request wait for the slowest one to finish, vLLM admits new sequences and evicts finished ones token by token, so a short request does not sit blocked behind a long one and the GPU never idles waiting on stragglers. Together these two ideas are what deliver the 10-to-24-times speed-up over a naive loop, and they are precisely why continuous batching is described as indispensable for shared online serving.

Installation is a single pip install, and the server launch is one command. This starts an OpenAI-compatible server on port 8000 for whichever open-weight model you have chosen:

# pip install vllm

# Launch an OpenAI-compatible server on port 8000.
# --max-model-len caps context; --max-num-seqs caps concurrency;
# --gpu-memory-utilization sets the VRAM slice for weights + KV cache.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --dtype auto \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 64 \
    --port 8000

Because vLLM speaks the OpenAI API, you do not need a bespoke client. The official OpenAI SDK works unchanged — you simply point base_url at your own server and pass any placeholder key. This is what makes vLLM a drop-in swap for a managed API in most codebases, and what lets you keep the same application code whether you are testing against a hosted model or your self-hosted one:

from openai import OpenAI

# vLLM speaks the OpenAI API, so the official client just works --
# point base_url at your server and use any placeholder api_key.
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarise this contract clause..."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

That is a complete, working serving stack: one command to launch, standard client to call. Everything from here is about making it fast enough and cheap enough to put in front of real users, and about not breaking it when you upgrade the model underneath.

Streaming tokens through a FastAPI wrapper

The single cheapest latency win available to you is not a faster GPU — it is streaming. Recall that TTFT is what the user waits on. If you buffer the entire generation and return it in one lump, the user stares at a spinner for the full response time. If you stream, they see the first token the instant it is ready and read along as the rest generates, so perceived latency collapses even though the total generation time is identical. For a chat box this is a pleasant improvement; for a realtime voice agent working to a strict latency budget with barge-in, streaming is not optional, because the whole interaction breaks if the first audio does not start almost immediately.

vLLM supports streaming natively through the same OpenAI-compatible interface — you pass stream=True and receive token deltas. In production you usually want your own thin edge in front of it, though: a FastAPI wrapper gives you a stable public contract, a place to attach authentication, rate limiting and request logging, and — importantly — a routing seam so you can swap the model behind the endpoint without changing the API your clients call. That seam is what lets you upgrade models behind a shadow and canary deploy rather than a risky straight-to-production edit. A minimal streaming wrapper forwards each token as it arrives:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI

app = FastAPI()
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

@app.post("/chat")
async def chat(prompt: str):
    stream = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )

    def token_generator():
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                yield delta      # forward each token the moment it arrives

    # text/event-stream keeps the connection open and flushes per token
    return StreamingResponse(token_generator(), media_type="text/event-stream")

Keep this layer genuinely thin. The one mistake that throws away the entire streaming benefit is buffering: if your wrapper, a proxy, or a load balancer collects the whole response before flushing it, you have paid for streaming and delivered a lump. Make sure every hop between the GPU and the browser is flush-per-token, and confirm it by measuring TTFT end-to-end from the client, not from the server.

Throughput versus latency: budgeting TTFT and TPOT

Now the central trade-off. TTFT decomposes into two parts: the prefill — the model processing your entire prompt before it can emit anything — plus any time the request spent waiting in the queue. TPOT is the per-step decode time, and it is dominated by how many sequences are sharing the GPU in the same batch. This is where throughput and latency pull against each other. Batching more requests together raises aggregate throughput — total tokens per second across all users, which is what determines your cost per token — but each additional concurrent sequence competes for compute and memory bandwidth, nudging TPOT up, and once requests start queuing, TTFT rises too.

The two extremes make the tension concrete. A batch of one — a single user with the whole GPU to themselves — gives the best possible TTFT and TPOT and the worst possible GPU utilisation and cost per token. A very deep batch gives peak throughput and the cheapest tokens, but the worst tail latency, because everyone is queuing and sharing. Neither extreme is right; your job is to find the point on that curve that meets your latency SLO at the lowest cost.

Crucially, with continuous batching you do not choose a fixed batch size at all. You cap concurrency with --max-num-seqs and let vLLM's scheduler fill the batch dynamically from whatever traffic is present. The right way to set that cap is to budget backwards from the user. Decide the TTFT and TPOT you can tolerate — for interactive chat, a TTFT under roughly half a second feels near-instant and a TPOT under roughly 50 milliseconds per token reads faster than most people can — then raise --max-num-seqs until you are sitting right at the edge of that budget under peak load. That edge is your throughput ceiling per GPU. A voice agent needs a far tighter budget and therefore a shallower batch; an overnight batch-analysis job can run the batch as deep as the hardware allows.

The table below shows the shape of the trade-off. The absolute figures are illustrative placeholders to convey direction, not measured results — always benchmark on your own model, GPU and prompt distribution, because the real numbers depend heavily on prompt length, output length and card:

The pattern the numbers gesture at is reliable even if the exact values are not: TTFT and TPOT both degrade gracefully as you deepen the batch, until the queue forms and TTFT climbs sharply. You want to operate just below that knee — deep enough to keep the GPU well utilised, shallow enough that a peak does not push you over the cliff.

Optimisations that actually move the needle

Once the basic serving stack is up and budgeted, three optimisations deliver most of the remaining wins, and NVIDIA's guide to mastering LLM inference optimisation singles out the same trio — FP8 quantisation, caching and speculative decoding — as the levers that cut latency. The order in which you adopt them matters.

FP8 quantisation is the first thing to try because it is the biggest, simplest win. Storing weights in eight-bit floating point roughly halves the weight memory versus FP16, which either lets you fit a substantially larger batch on the same card — directly raising throughput — or lets a bigger model fit where it previously would not. As of mid-2026, quality loss is minimal on most models, but "most" is not "all", and quantisation is exactly the kind of change that can quietly shift outputs on the long tail. Validate it on your evaluation set and ramp it behind the same shadow-and-canary discipline you would use for a model swap. This is also where quantisation-aware training pays off: models like Gemma 4 with QAT are trained to stay robust under low-bit inference, so they tolerate aggressive quantisation with far less regression than a model quantised naively after the fact.

Prefix and KV caching is the next lever, and it is enormous for the right workload. If many of your requests share a long common prefix — a large system prompt, a retrieval context block, a lengthy few-shot preamble — vLLM's automatic prefix caching reuses the computed KV cache for that shared prefix across requests instead of recomputing it every time. Because the prefill of that shared block dominates TTFT, caching it slashes time-to-first-token for every request that hits it. Enable it with --enable-prefix-caching. The economics are conceptually identical to provider-side prompt caching that cuts LLM costs across Claude, GPT and Gemini — you are paying once to process a prefix that many requests share — and it helps most for retrieval-augmented generation and for agent loops with a stable preamble.

Speculative decoding is the third lever and the most situational. A small, fast draft model proposes several tokens ahead, and the large model verifies them all in a single forward pass; when the draft guessed right, you get multiple tokens for the cost of one big-model step, cutting TPOT. It helps most when the draft agrees often — predictable, structured or low-entropy output such as code, JSON or templated text. On highly creative, high-entropy generation where the draft is usually wrong, the verification overhead can make it a net loss. Reach for it only after quantisation and caching, and only if TPOT is still your bottleneck.

Autoscaling in production, across Mumbai and London

The most common and most expensive autoscaling mistake with LLM inference is scaling on the wrong signal. Teams reach for the defaults their platform hands them — CPU and memory utilisation — and those defaults are actively wrong for GPU serving. An inference replica can be pegged at 100 per cent GPU with an almost idle CPU, so CPU utilisation tells you nothing about whether users are waiting. NVIDIA's production-serving guidance is clear on this point: autoscale on queue depth, GPU utilisation and P95 latency — not CPU or memory.

Each of those three signals plays a role. Queue depth — the number of pending requests per replica — is your leading indicator; it rises before latency does, so scaling on it lets you add capacity before users feel pain. GPU utilisation tells you whether the replicas you already have are genuinely busy, which stops you scaling out when the real problem is elsewhere. P95 or P99 latency against your SLO is the lagging alarm — by the time it fires, users are already affected, so it is a backstop, not a primary trigger. Scale out when queue depth or P95 crosses a threshold; scale in cautiously, because GPUs are expensive to thrash and slow to warm.

That warmth point deserves emphasis. Loading model weights onto a GPU can take tens of seconds, so scale-to-zero — attractive on paper for cost — adds a brutal cold-start penalty to the first request after a quiet period. For anything latency-sensitive, keep a warm floor of replicas always running, pre-pull your container images, and cache weights on the node so a scale-up event does not also pay a fresh download. The cost of a small warm floor is almost always less than the cost of cold-start latency in lost users.

Geography is the other half of production serving, and it is where the dual-market reality bites. Serve Indian users from AWS Mumbai (ap-south-1) and UK and European users from London (eu-west-2). This cuts network round-trip time — a real and often large component of TTFT for distant users — and keeps data in-region, which matters for India's DPDP expectations and UK GDPR alike. Route by geography at the edge, and hold a separate latency baseline per region, because "within budget" is a local number shaped by where your users and your inference endpoints physically sit; do not import London's threshold into Mumbai or vice versa. And when you roll a new model, a new quantisation or a new region live, ramp it behind shadow traffic and a canary rather than flipping every region at once — the serving discipline is the same whichever city the GPU lives in.

Pitfalls and next steps

The failures that catch self-hosting teams are boringly consistent, which is good news — you can pre-empt every one of them:

1. Setting --gpu-memory-utilization too high. A value that works on quiet traffic tips into an out-of-memory crash the moment a long-context spike arrives. Leave headroom and validate the cache under peak load.
2. Deep batches that flatter the throughput dashboard. A configuration that looks glorious on tokens-per-second can be blowing your tail latency to pieces. Budget from the user's TTFT and TPOT, never from the GPU's peak.
3. Shipping a quantised or swapped model with no eval gate. Silent, distributional quality regressions are the signature failure of an un-gated model change; ramp them through shadow and canary deploys with pre-committed rollback triggers.
4. Autoscaling on CPU or memory. Scale on queue depth, GPU utilisation and P95 latency instead, or you will add capacity that does not help and fail to add capacity that does.
5. Ignoring cold starts. Scale-to-zero that adds tens of seconds to the first request after a lull will quietly punish exactly the users returning to your product. Keep a warm floor.
6. Forgetting data residency. The cheapest region is not always a lawful region for your users' data — weigh DPDP and UK GDPR expectations alongside the GPU-hour price.

The path forward is incremental and measurable. Start with a single 24 GB GPU and the smallest open-weight model that passes your evaluation set. Stand up the vLLM server, put a thin streaming FastAPI edge in front of it, and measure real end-to-end TTFT and TPOT under representative load before you touch a single batch-size knob. Add prefix caching if your prompts share a common preamble; reach for FP8 quantisation next, gated by evals; and only add speculative decoding if TPOT is still your bottleneck. Wire autoscaling to queue depth and P95, keep a warm floor, and split serving between Mumbai and London for your two markets. Put the whole thing behind a routing layer you can flip so every future upgrade rides a shadow-and-canary ramp rather than a leap of faith.

Self-hosting an open LLM on vLLM is no longer exotic. The vllm serve command is trivial; the craft is in the budgeting and the discipline — knowing which two numbers you are optimising, sizing the KV cache honestly, gating your optimisations, and scaling on signals that actually reflect whether users are waiting. Get that right and a self-hosted model becomes a predictable, controllable, cost-effective serving tier — the same on a Tuesday in Bengaluru as on a Tuesday in London.