Load-testing and capacity planning for LLM apps: the metrics that actually matter

What you will actually measure

Most teams load-test their large language model service the way they load-test a REST API: point a tool at the endpoint, ramp up virtual users, watch requests-per-second and average response time, and call it capacity. Then production traffic hits, the first token takes four seconds to appear during the evening peak, and nobody can say why — because the dashboard was measuring the wrong things all along.

LLM serving does not behave like a stateless CRUD endpoint. Responses stream token by token rather than arriving in one block. Output length varies enormously from one request to the next. The meaningful signals live at the token level, not the request level. And the hardware saturates in a pattern that has more to do with GPU memory and a structure called the KV cache than with CPU or network. If your load test does not account for all four of those, the numbers it produces are, at best, decorative.

This guide is the field manual I wish I had handed my own team before our first serious traffic spike. It covers the metrics worth watching, how to pick a load-testing tool, how to write a test that drives realistic streaming traffic and measures time-to-first-token honestly, how to read the results without fooling yourself, and how to turn all of that into a defensible capacity plan using Little's Law. The examples assume you are serving from a region close to your users — AWS Mumbai for an Indian audience, London for a UK one — because latency budgets and GPU pricing both change with geography.

Why requests-per-second lies for LLMs

Start with the single most important idea in this article: tokens per second, not requests per second, is the real throughput metric. Requests-per-second is a misleading proxy because output length varies. One user asks for a yes-or-no classification and receives twelve tokens. The next asks for a full incident report and receives twelve hundred. Requests-per-second scores those as identical units of work when the second consumed roughly a hundred times more decode compute. A service happily sustaining 50 requests per second of short classifications will fall over at 50 requests per second of long generations, and the requests-per-second chart will look flat and healthy right up to the moment it does.

The deeper reason is that an LLM request is not one operation, it is two phases with completely different performance characteristics. The prefill phase reads the entire prompt and builds the internal state in one compute-heavy burst; this is what dominates time-to-first-token. The decode phase then generates output one token at a time, each step depending on the last, which is memory-bandwidth-bound rather than compute-bound. A load test that reports a single average latency number smears these two phases together and tells you nothing actionable about either.

There is a third divergence from ordinary REST testing: GPU saturation patterns. A web server degrades gracefully as CPU climbs toward 100 per cent. An LLM server behind a modern engine batches many sequences together and shares GPU memory between them via the KV cache. When that cache fills, the server does not slow down smoothly — it starts queuing or pre-empting requests, and TTFT jumps sharply. You need to test right up to and past that cliff to know where it is. If you have already read our companion piece on self-hosted LLM serving, quantisation and batching, this is the operational other half of that story: batching decides your efficiency, load-testing tells you where it breaks.

The metrics that matter

Before you write a single line of test script, agree on the metrics you will hold your service to. These are the six that a streaming LLM service actually lives and dies by. Note the split between prefill signals and decode signals — conflating them is the classic mistake.

Metric	What it measures	Phase / signal	Why it matters
TTFT — time-to-first-token	Delay from request sent to first token received	Prefill signal	Governs perceived responsiveness; the number users feel first
TPOT — time-per-output-token	Average gap between consecutive output tokens (inter-token latency)	Decode signal	Sets how fast the answer streams once it has started
Output tokens/sec	Generation rate per individual stream	Decode throughput	The honest per-user throughput figure, immune to output-length skew
p99 TTFT under concurrency	99th-percentile first-token latency at target load	Prefill tail	Exposes the queuing cliff; where your SLA quietly breaks
5xx rate at the spike	Share of failed responses at peak concurrency	Reliability	Tells you whether headroom is real or you are shedding load
Cost per test run	GPU-hours consumed to produce the result	Economics	Ties every capacity decision back to the monthly bill

A useful mental model: TTFT and p99 TTFT tell you whether the service feels fast to a person waiting for a chat reply; TPOT and output tokens/sec tell you whether the service is productive for an agent or batch job that consumes the whole response. A retrieval-augmented assistant with a long system prompt might have a poor TTFT but excellent TPOT, while a short-prompt code helper might be the reverse. Optimise for the one your product depends on. If your product streams to a browser, our guide on streaming LLM responses at scale with SSE and backpressure covers the transport-layer half of keeping TTFT low.

Choosing a load-testing tool

There is no single right tool, only trade-offs between the language your team already knows, how many concurrent streaming connections you need to sustain, and how much visibility you want into the serving engine itself. Here is how the main options compare for LLM work specifically.

Tool	Language / model	Streaming (SSE) support	Token-level metrics	Best for
k6	JavaScript scenarios, Go engine; built-in gRPC	Yes, HTTP + SSE	TTFT / ITL via custom Trend metrics	High concurrency, mixed HTTP and gRPC, CI pipelines
Locust	Python, single-threaded event loop per worker	Yes, with manual stream handling	Custom via events.request hooks	Python-native teams; needs distributed workers at scale
vLLM benchmark suite	Python, ships with the serving engine	Yes, native to the engine	TTFT, TPOT, throughput plus KV-cache utilisation via Prometheus	Tuning a self-hosted vLLM deployment end to end
GenAI-Perf	Python, from the Triton ecosystem	Yes	Rich token-level and concurrency sweeps out of the box	Standardised, repeatable inference benchmarks
llmperf / LLM-Locust	Python, LLM-specific harnesses	Yes	Purpose-built TTFT / ITL / tokens-per-sec	Quick apples-to-apples model or provider comparisons

Two practical notes. First, k6 handles server-sent events and thousands of concurrent long-lived connections comfortably because its execution engine is compiled Go, but it does not know what a token is — you must add custom metrics to capture TTFT and inter-token latency yourself. Second, Locust is a joy if your team writes Python, yet its single-threaded event loop becomes the bottleneck when you simulate thousands of long-lived SSE connections; you end up measuring the load generator rather than the service. If you go with Locust at real scale, run it in distributed mode with many worker processes so no single event loop is saturated.

Writing a realistic test

The test below is a k6 scenario that talks to an OpenAI-compatible streaming endpoint, measures TTFT precisely as the delay to the first content token, tracks inter-token latency, and computes output tokens per second per stream. Crucially, it draws prompts from a pool of variable lengths rather than repeating one fixed string — because fixed lengths are the single biggest source of measurement bias, a point we return to below. k6 does not expose SSE natively as a first-class API in every version, so this reads the streamed body incrementally over plain HTTP, which is portable and transparent about exactly what it measures.

import http from 'k6/http';
import { Trend, Rate } from 'k6/metrics';
import { SharedArray } from 'k6/data';

// Custom, token-level metrics — k6 does not know what a token is.
const ttft      = new Trend('llm_ttft_ms', true);   // prefill signal
const tpot      = new Trend('llm_tpot_ms', true);   // decode signal
const tokPerSec = new Trend('llm_output_tps');      // per-stream throughput
const errRate   = new Rate('llm_5xx_rate');

// Variable-length prompts — NEVER a single fixed string.
const prompts = new SharedArray('prompts', () =>
  JSON.parse(open('./prompts.json')));   // mix of short, medium, long

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 20 },   // warm the KV cache
        { duration: '3m', target: 100 },  // find the knee
        { duration: '2m', target: 200 },  // push past saturation
        { duration: '1m', target: 0 },
      ],
    },
  },
  thresholds: {
    'llm_ttft_ms': ['p(99)<3000'],   // p99 first token under 3s
    'llm_5xx_rate': ['rate<0.01'],
  },
};

export default function () {
  const prompt = prompts[Math.floor(Math.random() * prompts.length)];
  const body = JSON.stringify({
    model: 'my-served-model',
    stream: true,
    max_tokens: 512,
    messages: [{ role: 'user', content: prompt }],
  });

  const start = Date.now();
  let firstTokenAt = null;
  let lastTokenAt  = null;
  let tokenCount   = 0;

  // Stream the response and time each SSE data frame as it lands.
  const res = http.post('https://llm.internal/v1/chat/completions', body, {
    headers: { 'Content-Type': 'application/json', 'Accept': 'text/event-stream' },
    responseType: 'text',
    timeout: '60s',
  });

  errRate.add(res.status >= 500);
  if (res.status >= 500) return;

  // Each 'data:' line is one streamed chunk (one or more tokens).
  const frames = String(res.body).split('\n').filter(l => l.startsWith('data:'));
  for (const frame of frames) {
    if (frame.includes('[DONE]')) break;
    const now = Date.now();
    if (firstTokenAt === null) {
      firstTokenAt = now;
      ttft.add(firstTokenAt - start);      // TTFT = prefill wait
    }
    lastTokenAt = now;
    tokenCount += 1;
  }

  if (tokenCount > 1 && lastTokenAt > firstTokenAt) {
    const decodeMs = lastTokenAt - firstTokenAt;
    tpot.add(decodeMs / (tokenCount - 1));            // inter-token latency
    tokPerSec.add((tokenCount / decodeMs) * 1000);    // output tokens/sec
  }
}

A few design choices are load-bearing. The ramp deliberately climbs past your expected peak so the test reveals the saturation cliff rather than stopping short of it. The warm-up stage exists because a cold KV cache and cold routing give an optimistic first minute that would pollute your percentiles. And the prompts come from a file with a realistic distribution of lengths — short questions, medium chats, and long retrieval-augmented contexts in the proportion your real traffic shows. If you serve a gateway in front of several models, pair this with the failover behaviour described in our note on building a resilient LLM gateway with failover, retries and rate limits, and run the test through the gateway, not around it, so retries and rate limits are part of what you measure.

Reading the results without fooling yourself

Once the run finishes, resist the urge to look at the mean. The mean is where LLM capacity planning goes to die. Look at the distribution: p50 versus p90 versus p99 TTFT. A healthy service at moderate load might show p50 TTFT of 350ms and p99 of 800ms — a tight spread. As you approach the saturation point, the p50 barely moves while the p99 balloons, because a growing fraction of requests are queuing behind a full batch. That widening gap between median and tail is the signal. Your capacity is the concurrency at which p99 TTFT breaches your SLA, not the concurrency at which the average does.

The second thing to read is KV-cache saturation. If you are self-hosting on vLLM, its benchmark suite and Prometheus metrics expose KV-cache utilisation directly — watch the fraction of cache blocks in use climb as concurrency rises. When it approaches 100 per cent, the engine can no longer admit new sequences into the running batch, so it queues them, and queued requests are exactly the ones whose TTFT explodes. Overlay the KV-cache utilisation curve on the p99 TTFT curve and you will usually see them turn upward at the same instant. That instant is your per-replica ceiling.

The third thing is the 5xx rate at the spike. A service that keeps p99 TTFT flat by returning 503s under load has not got more capacity — it is shedding work. Always read the tail latency and the error rate together. Headroom is only real if both stay within budget at your target concurrency. Speculative decoding can buy back some of that tail; if decode latency is your constraint, our piece on speculative decoding for faster LLM inference is the next lever to pull once your load test has proven decode is the bottleneck.

Capacity planning with Little's Law

Now turn measurements into a plan. The tool for this is Little's Law, a result from queuing theory that is almost embarrassingly simple and almost always right:

average concurrency ≈ arrival rate × average request duration

Suppose your Indian consumer app sees a peak of 30 requests per second, and the average request — prefill plus a 512-token generation — takes 4 seconds end to end. Then your average in-flight concurrency is 30 × 4 = 120 concurrent sequences. That is the number you must serve without breaching your TTFT budget. From the load test you know that a single replica keeps p99 TTFT under target up to, say, 45 concurrent sequences before the KV cache saturates. So you need at least 120 ÷ 45 ≈ 3 replicas to carry the mean, and you add headroom on top for variance, deploys, and traffic bursts — call it 4 or 5. This is the same reasoning whether you serve from Mumbai or London; only the arrival rate and the GPU price change.

Two forces sit underneath Little's Law and deserve respect. The KV cache bounds how many sequences can run concurrently on one GPU — it is the physical limit that your safe per-replica concurrency figure encodes. And continuous batching raises GPU utilisation by slotting new sequences into the batch as others finish, rather than waiting for the whole batch to complete; it is what lets a single replica hold 45 sequences instead of a handful. Load-testing tells you the real per-replica number after batching; Little's Law tells you how many of those replicas your traffic demands.

Finally, price the plan. In mid-2026, an H100 SXM rents for roughly $2.00 to $2.50 per GPU-hour on specialised GPU clouds, and up to $6.50 or more per hour on the large hyperscalers. Five replicas on one H100 each, at $2.25 per GPU-hour on a specialised cloud, is about $270 a day; the same footprint on a hyperscaler could be three times that. Size your headroom against that spread deliberately — the gap between a comfortable p99 and a reckless one might be one extra GPU, and whether that GPU costs $54 or $156 a day is a real business decision, not a rounding error.

Common mistakes to avoid

Almost every misleading LLM load test I have reviewed made at least one of a short list of errors. The most damaging is measurement bias from fixed lengths: fixed prompt and output lengths produce uniform prefill and decode work, the KV cache fills predictably, and tail latency looks artificially flat. Real traffic has variable-length prompts and outputs that create uneven cache pressure and genuine long tails. If your test uses one prompt repeated, your p99 is fiction. Drive realistic, variable-length traffic — the SharedArray pool in the script above exists precisely for this.

The rest of the list is shorter but no less common: reading the mean instead of the p99; measuring requests-per-second instead of tokens-per-second; running the load generator too small so it, not the GPU, is the bottleneck; testing against a warmed single request and never pushing past the saturation cliff; and forgetting to test through the real gateway with its retries and rate limits in place. Get those right and your load test stops being theatre and starts being the thing that lets you promise an SLA and keep it.

Reliable inference is not a lucky accident; it is the visible output of a team that measured the right things and sized the fleet honestly. That work is exactly the kind of thing worth showing on a profile where the people hiring can see it.