Autoscaling LLM Inference on Kubernetes with KServe and KEDA

What you need to know

Kubernetes has quietly become the default place teams run LLM inference. According to CNCF's 2025 Annual Cloud Native Survey, 66% of organisations hosting generative AI models use Kubernetes to manage some or all of their inference workloads — the majority, and rising. That matters because the autoscaler most teams reach for first, the standard Horizontal Pod Autoscaler wired to CPU and memory, is built for the wrong bottleneck. A GPU inference pod can sit at effectively 100% GPU while its CPU idles in single digits, so a CPU-based autoscaler either never fires or fires on noise that has nothing to do with whether users are waiting.

• CPU and memory are the wrong signal. GPU inference is bound by GPU compute and memory bandwidth, not CPU cycles — scale on what the GPU and the request queue are actually doing.
• Four signals do the real work: request queue depth, GPU utilisation from a metrics exporter, requests or tokens in flight, and P95/P99 latency against your SLO.
• KServe and KEDA are complementary, not competing. KServe gives you a standard InferenceService for your model server; KEDA gives that service event-driven scaling — including scale-to-zero — from the metrics above.
• Scale-to-zero saves real money but has a genuine cold-start cost that ranges from single-digit seconds to several minutes depending on what, exactly, was scaled to zero.
• Quantisation, spot capacity and right-sizing compound — used together they routinely cut serving cost well below what any single lever achieves alone, provided you gate each change with evals.

Why CPU and memory autoscaling fails for GPU inference

The default Kubernetes autoscaling story is a CPU-shaped one. The Horizontal Pod Autoscaler was designed for web services where CPU utilisation tracks load reasonably well, and for years that assumption held because most workloads on the cluster were, in fact, CPU-bound. LLM inference breaks the assumption cleanly. The expensive part of serving a request — the forward pass through billions of parameters — happens almost entirely on the GPU. The pod's CPU is mostly doing bookkeeping: tokenising input, marshalling requests, streaming output. It can sit at 5–10% utilisation while the GPU underneath it is fully saturated and users are queuing.

This is not a niche observation — it is the explicit, repeated guidance from every major cloud provider's Kubernetes documentation. Google Kubernetes Engine's best-practice guide for autoscaling LLM inference workloads is explicit that CPU and memory utilisation should not be used as the only indicator of GPU-backed inference load, because these workloads primarily consume GPU resources and CPU-only autoscaling leads to both wasted spend and missed capacity. AWS's guidance for EKS says the same for GPU model-serving workloads, and Microsoft's Azure Kubernetes Service documentation goes further still, recommending KEDA drive scaling from inference-server latency metrics rather than resource utilisation at all. Three separate hyperscalers converging on the same warning, independently, is about as strong a signal as infrastructure advice gets.

The fix is not simply "watch the GPU instead of the CPU" — that trades one blunt instrument for a slightly sharper one. GPUs expose their own telemetry through NVIDIA's Data Center GPU Manager (DCGM) exporter, most commonly the DCGM_FI_DEV_GPU_UTIL metric scraped into Prometheus. But Microsoft's own AKS documentation carries an important caveat: do not treat DCGM_FI_DEV_GPU_UTIL as a precise efficiency score, because it only reports whether a GPU kernel was active during the sampling window, not how much useful work it did. Because vLLM and similar servers use continuous batching to keep the GPU busy across a wide range of concurrency, utilisation can read close to 100% whether you are serving five requests or fifty — which is a version of the same problem CPU metrics have, one level down the stack. That is why the next section leans on request-level signals as the primary trigger and GPU-level metrics as a supporting one, not the other way round.

The metrics that actually predict GPU inference load

Once you accept that CPU and memory are out, four signals cover almost everything you need, and each plays a different role in the decision. Request queue depth — the count of requests waiting to be picked up by the model server, exposed by vLLM as vllm:num_requests_waiting — is the leading indicator. It rises before latency does, which is exactly what you want from a trigger: it gives the autoscaler time to bring up a new replica before users feel anything. GKE's own autoscaling guidance names queue size as the primary metric for throughput-sensitive workloads for precisely this reason, and also highlights a purpose-built alternative for LLM serving specifically — KV cache utilisation, exposed through GKE's Inference Gateway as inference_pool_average_kv_cache_utilization — which tracks how full the model server's memory actually is rather than how many requests are stacked up.

Requests or tokens in flight — vLLM's vllm:num_requests_running — tells you how many sequences the server is actively decoding right now, as distinct from how many are waiting. This is your batch-depth proxy: pair a target value on this metric (KServe's own OpenTelemetry example uses a target of four running requests per pod) with a queue-depth trigger and you get both "am I about to fall behind" and "am I already at capacity" in one configuration. GPU utilisation from DCGM plays a supporting role — it confirms whether replicas you already have are genuinely busy, which stops you scaling out when the real problem is somewhere else in the request path (a slow tokenizer, a downstream dependency, a network hop). And P95 or P99 latency against your service-level objective is the lagging alarm: by the time it fires, some users have already had a slow response, so treat it as a backstop and an alerting threshold rather than your primary trigger. Microsoft's AKS guidance leans on TTFT P95 specifically as a KEDA scaling signal for exactly this reason — it is the number closest to what a user actually experiences.

KServe and KEDA: how the pieces fit together

KServe is the serving layer: it wraps a model server — vLLM, Triton, Hugging Face TGI, or a custom runtime — in a standard InferenceService custom resource, giving you consistent routing, versioning, canarying and an autoscaling interface regardless of which server sits underneath. Left to its defaults, KServe's "Standard" deployment mode falls back to the same CPU-and-memory HPA behaviour described above, which is not very useful for a GPU workload. That is the gap KEDA closes. KServe's own documentation describes the integration directly: set the annotation serving.kserve.io/autoscalerClass: "keda" on an InferenceService and KServe creates a KEDA ScaledObject behind the scenes, which itself manages an HPA — but one driven by whatever external metric you configure, including Prometheus queries and OpenTelemetry-collected pod metrics, rather than CPU or memory alone.

This combination unlocks two things vanilla KServe autoscaling cannot do on its own. First, LLM-native metrics: KServe's generative-inference autoscaling documentation shows both a Prometheus-scraping path and an OpenTelemetry sidecar-injection path for pulling vLLM's own metrics — queue depth, requests running — straight into the scaling decision. Second, genuine scale-to-zero: KEDA supports scaling a Deployment to zero replicas when there is no traffic, which the standard HPA cannot do at all. For a GPU pool, where every idle replica is burning an expensive GPU-hour for nothing, that second capability is where most of the cost saving actually lives.

The illustrative configuration below shows the shape of a KEDA-driven InferenceService for a vLLM-backed model, scaling on queue depth via an OpenTelemetry-collected metric, with scale-to-zero enabled through minReplicas: 0. Treat the exact field names as indicative — always check them against your installed KServe and KEDA versions before shipping:

# Illustrative — verify field names against your KServe/KEDA versions.
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama-8b-instruct
  annotations:
    serving.kserve.io/deploymentMode: "Standard"
    serving.kserve.io/autoscalerClass: "keda"
    sidecar.opentelemetry.io/inject: "llama-8b-instruct-predictor"
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "hf://meta-llama/Llama-3.1-8B-Instruct"
      runtime: kserve-vllmserver
    resources:
      limits:
        nvidia.com/gpu: "1"
    # minReplicas: 0 enables scale-to-zero on this pool.
    # Set to 1+ for latency-sensitive traffic that cannot absorb a cold start.
    minReplicas: 0
    maxReplicas: 8
    autoScaling:
      metrics:
        # Leading signal: requests waiting to be picked up.
        - type: PodMetric
          podmetric:
            metric:
              backend: "opentelemetry"
              metricNames:
                - vllm:num_requests_waiting
              query: "vllm:num_requests_waiting"
            target:
              type: Value
              value: "4"
        # Secondary signal: requests already being decoded (batch depth proxy).
        - type: PodMetric
          podmetric:
            metric:
              backend: "opentelemetry"
              metricNames:
                - vllm:num_requests_running
              query: "vllm:num_requests_running"
            target:
              type: Value
              value: "8"

One operational rule matters more than any single field in that manifest: never point a manual HPA and a KEDA ScaledObject at the same workload. Microsoft's AKS documentation flags this explicitly — because KEDA manages its own HPA under the hood, a second, independently configured HPA on the same Deployment fights it for control and produces genuinely strange scaling behaviour. Let KEDA own the whole decision once you have adopted it. If your team already has strong opinions on self-hosted serving fundamentals — GPU sizing, KV-cache budgeting, continuous batching — our vLLM throughput and latency playbook covers the serving layer this autoscaling sits on top of in more depth.

Scale-to-zero: when it pays off and when it costs you

Scale-to-zero is the single biggest lever for eliminating idle-GPU cost, and idle GPU spend is not a marginal problem — industry estimates put a very large share of provisioned AI compute sitting idle at any given moment, and every idle GPU-hour on a dedicated pool is money spent serving nobody. But "scale to zero" hides two very different operations, and conflating them is where teams get burned on latency.

The first is scaling pods to zero on a GPU node pool that stays provisioned and warm. Here the cold start is just the cost of loading model weights onto a GPU that is already up, drivers already installed, image already local. From local NVMe storage, that is commonly single-digit seconds for a small quantised model and well under a minute even for larger checkpoints — vLLM itself adds a few seconds of initialisation on top for setting up its PagedAttention memory pools and warming the CUDA context. The second is scaling the node pool itself to zero. Now the first request after a lull also waits on a brand-new node: capacity provisioning, GPU driver installation, and pulling your (likely multi-gigabyte) serving image, before model loading even starts. Microsoft's AKS documentation puts that full sequence at roughly five to ten minutes end to end — node provisioning alone accounts for three to five of those minutes.

That gap — seconds versus minutes — is the whole decision. For internal tools, staging environments, batch or overnight jobs, and genuinely spiky low-traffic endpoints, node-level scale-to-zero is close to free money: nobody is watching a spinner during the rare cold start. For anything user-facing and latency-sensitive, the right pattern is to keep a warm floor of GPU nodes always running — even one or two — and scale only the KServe pods to zero on top of that warm capacity. You still eliminate the majority of idle-GPU spend during quiet hours, but the worst-case first request is seconds, not minutes.

Cutting cost further: quantisation, spot instances and right-sizing

Autoscaling controls how many GPUs you run; these three levers control how expensive each GPU-hour is and how much it can do. Used together, and gated properly, they compound.

Quantisation shrinks the weights themselves. FP16 is the common baseline at 2 bytes per parameter. Dropping to FP8 — 1 byte per parameter — roughly halves weight memory, is natively supported by Hopper- and Blackwell-class Tensor Cores, and carries minimal quality loss on most current models, which is why it is usually the first optimisation worth trying. Going further to INT4 formats such as AWQ or GPTQ gets you to roughly 0.5 bytes per parameter — about a quarter of FP16 — freeing substantial VRAM for a bigger batch or a larger KV cache, at the cost of a larger, though for most models still moderate, quality hit; AWQ's activation-aware calibration tends to hold accuracy slightly better than plain GPTQ at the same bit width. None of these are free lunches — treat every quantisation step as a model change that needs an eval gate, exactly as our vLLM playbook argues for FP8 in isolation.

Spot and preemptible GPU capacity attacks the price of the GPU-hour itself rather than how much a GPU-hour can do. AWS advertises Spot Instance savings of up to 90% off on-demand pricing, and Google Cloud and Azure publish broadly comparable ranges for their preemptible and spot tiers; realised, sustained savings on GPU capacity specifically are more commonly cited in the 50–70% range once you account for interruption handling, checkpoint overhead and the occasional need to fall back to on-demand. The catch is obvious but easy to under-engineer for: spot capacity can be reclaimed with short notice, so it only belongs on replicas your autoscaler can lose without dropping a request in flight — the elastic portion of your fleet that KEDA scales up and down with traffic, never the warm floor that must always answer. Pair spot replicas with request draining on the reclaim signal and an automatic fallback to on-demand capacity if the spot pool empties out.

Right-sizing is the least glamorous lever and often the largest one: pick the smallest GPU generation and the smallest model that clears your latency and quality bar, rather than defaulting to whatever the team used last time. Newer GPU generations frequently change this calculus meaningfully — the shift from H100 to B200-class hardware, for instance, has moved the cost-per-token maths enough to be worth re-running your sizing exercise every time a new generation reaches general availability, rather than assuming last year's hardware choice still holds.

Common pitfalls when autoscaling GPU inference

1. Leaving the default CPU/memory HPA in place "for now." It is the single most common mistake, and it either silently under-scales through real GPU saturation or churns replicas on irrelevant CPU noise. Replace it before you go anywhere near production traffic.
2. Running a manual HPA alongside a KEDA ScaledObject on the same Deployment. They compete for control of the same underlying HPA object and produce erratic, hard-to-debug scaling. Pick one owner.
3. Treating GPU utilisation as a saturation score. A DCGM reading of "100%" tells you the GPU was busy, not how much headroom is left — pair it with queue depth before acting on it.
4. Scaling latency-critical endpoints to zero at the node level. A five-to-ten-minute cold start on a user-facing path is an outage by another name. Reserve full node-level scale-to-zero for staging, batch and genuinely tolerant traffic.
5. Shipping a quantised model with no eval gate. FP8 and INT4 are behavioural changes, not free performance wins — run them through the same shadow-and-canary discipline you would use for any model upgrade before they touch real traffic.
6. Putting your warm floor on spot capacity. The elastic burst layer is fine on spot; the baseline that must always answer a request is not — a reclaim event on your floor is a self-inflicted outage.
7. Ignoring where the GPU actually sits. Serve Indian users from a Mumbai region and UK/EU users from a London region for both latency and data-residency reasons — DPDP and UK GDPR expectations do not disappear because a spot GPU was cheaper two regions away. A resilient front door matters here too; see our guide to building a resilient LLM gateway for the failover and rate-limiting layer that should sit in front of an autoscaled backend.

A worked example: autoscaling a mid-size deployment

The following numbers are clearly illustrative — a composite of typical patterns rather than a single measured deployment — but they show how the levers above stack in practice. Picture a mid-size product serving an 8-billion-parameter instruction model to a few hundred concurrent users across India and the UK, averaging perhaps 80–150 requests per minute with a sharp midday and evening peak in each region.

A naive always-on baseline sized for peak — say eight always-on GPU replicas to cover the worst hour of the day — pays for eight GPU-hours every hour of the day, whether or not anyone is using the product. A KServe-and-KEDA setup driven by queue depth might instead run a warm floor of two on-demand replicas around the clock, scaling out to eight during peak windows using a mix of on-demand and spot capacity for the elastic replicas, and scaling a separate staging InferenceService fully to zero outside working hours. Layering FP8 quantisation on top roughly halves the weight memory footprint, which in this illustrative scenario is enough to lift the batch size the same two-replica floor can serve before it needs to scale out at all, delaying — and therefore reducing the frequency of — every subsequent scaling event.

Put together, illustratively: a fixed eight-replica baseline running 24 hours a day represents roughly 192 GPU-hours per day. A floor of two replicas plus autoscaled peak capacity averaging perhaps three additional replicas across the peak windows, with FP8 quantisation reducing how often those peaks are hit at all, might land closer to 90–110 GPU-hours per day for the same traffic — before spot pricing on the elastic replicas is even applied. Layer in a 50–70% discount on just the elastic portion of that capacity and the daily bill for GPU-hours alone can fall by roughly half versus the always-on baseline, for materially the same user-facing latency. Treat every figure in this paragraph as an illustration of the shape of the saving, not a number to copy into a budget — your own traffic curve, model size and region mix will move it substantially in either direction, which is exactly why the earlier sections insist on measuring your own queue depth, latency and utilisation before committing to thresholds.

Where to take this next

None of the individual pieces here are exotic — KServe, KEDA, DCGM and quantisation are all mature, widely documented tools by mid-2026. The craft is in the ordering: replace CPU/memory autoscaling first, because it is actively wrong; add queue-depth-driven KEDA scaling next; separate your warm floor from your scale-to-zero pool before latency-sensitive traffic ever touches it; and only then reach for quantisation and spot capacity to bring the cost of each GPU-hour down further. Get that sequence right and a Kubernetes-native LLM serving stack becomes a genuinely elastic, cost-proportionate piece of infrastructure — not a fixed GPU bill you happen to run inference workloads on top of.

If you have built and run this kind of autoscaling setup in production — on GKE, AKS, EKS or bare-metal Kubernetes, across a Mumbai or London region or both — that is exactly the sort of concrete, load-bearing work worth putting in front of the people hiring for it.