How do you run LLM inference on Kubernetes?
LLM inference on Kubernetes runs a model server, such as vLLM or SGLang, in Pods that request GPUs, behind a gateway that routes requests to replicas based on their load. Replicas scale on queue depth or KV-cache use rather than CPU, and model weights load from a volume, object storage or a cache because they are tens to hundreds of gigabytes. The rest is standard Kubernetes: Deployments, Services, node pools, taints and autoscalers.
The layers of an inference stack
| Layer | What it does | Examples |
|---|---|---|
| GPU nodes | Run the model servers on the right GPU type | Node pools with GPU taints, the NVIDIA device plugin |
| Model server | Loads the model, batches requests, manages the KV cache | vLLM, SGLang, NVIDIA Triton Inference Server, Hugging Face TGI |
| Serving layer | Deploys and scales models as Kubernetes resources | KServe, llm-d |
| Routing | Sends each request to the replica best placed to serve it | Gateway API Inference Extension |
| Autoscaling | Adds and removes replicas and GPU nodes | Horizontal Pod Autoscaler with custom metrics, KEDA, node autoscaling |
Model servers do the heavy lifting
A model server is the process that holds the weights in GPU memory and generates tokens. The difference between a naive server and a modern one is mostly in how they use that memory. vLLM and SGLang add new requests to a running batch as others finish (continuous batching), store the KV cache in pages so memory is not wasted on padding, and reuse cached prefixes such as a shared system prompt. Together these let one GPU serve many concurrent users. Both expose an OpenAI-compatible HTTP API.
A minimal deployment is a single container:
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-8b
spec:
replicas: 2
selector:
matchLabels:
app: llama-8b
template:
metadata:
labels:
app: llama-8b
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: vllm
image: vllm/vllm-openai:latest
args: ["--model", "meta-llama/Llama-3.1-8B-Instruct", "--max-model-len", "8192"]
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: 1Pin an exact image tag in production. Before choosing the GPU, work out whether the model and its KV cache fit, as shown in how much GPU memory an LLM needs.
Serving layers on top
Running one Deployment per model works for a few models. A serving layer adds a model-level API, rollouts and autoscaling. KServe, a CNCF project, offers vLLM and llm-d as backends for LLM serving with an OpenAI-compatible protocol. llm-d, a CNCF Sandbox project, runs vLLM and SGLang across a cluster and adds patterns such as prefill and decode on separate Pods and prefix-cache-aware routing.
Model-aware routing
A standard Kubernetes Service balances requests round robin. For LLMs that is a poor fit: one replica may hold a long queue while another sits idle, and a request whose prompt prefix is already cached on one replica is cheaper to serve there.
The Gateway API Inference Extension, an official Kubernetes project, turns a Gateway API gateway into an inference gateway. An InferencePool groups the model server Pods, and an endpoint picker reads metrics from each Pod, such as prefix cache status and loaded LoRA adapters, to decide which one gets each request. Envoy Gateway, kgateway and GKE Gateway implement it. If you are still on Ingress, Ingress vs Gateway API covers the move.
Autoscale on the right signal
CPU use says little about a GPU model server. Scale replicas on what limits throughput:
- Queue depth. vLLM exports
vllm:num_requests_waiting. A growing queue means replicas are saturated. - KV-cache use.
vllm:kv_cache_usage_percis the fraction of KV-cache blocks in use. Near 1, new requests wait for memory. - Latency. Time to first token and time per output token, measured at the gateway.
The Horizontal Pod Autoscaler can use these through a Prometheus metrics adapter, or KEDA can scale on a Prometheus query directly. When a new replica has no GPU to land on, node autoscaling adds one. How Kubernetes autoscaling works covers how these layers interact.
Plan for cold starts
A new replica is ready only when its node is up, the image is pulled and the weights are in GPU memory. Weights are the slow part: a 70B model in BF16 is about 140 GB. Common ways to load them:
| Source | Trade-off |
|---|---|
| Download from a model hub at startup | Simple, slowest, depends on an external service |
Pre-populated PersistentVolume | Fast reloads in one zone; volume must be readable by many Pods |
| Object storage in the same region | Scales well; throughput depends on parallel reads |
| Weights baked into a container image | Uses the image cache on each node; very large images |
Whatever the source, keep at least one warm replica for user-facing models, because scaling from zero takes minutes.
Models that need several GPUs
When a model does not fit on one GPU, the model server splits it. Tensor parallelism within one node is a single Pod that requests several GPUs, for example nvidia.com/gpu: 4 with vLLM’s --tensor-parallel-size 4, and works best on NVLink-connected GPUs. Models larger than one node need a group of Pods that start and fail together, which the Kubernetes LeaderWorkerSet API provides.
Where the GPUs come from
Inference capacity should sit close to users and data, and GPU availability differs by region and provider. Keep model servers on dedicated, tainted GPU node pools, share small models on one GPU where isolation allows (see GPU sharing), and spread replicas across zones or providers so that one sold-out region does not stop scaling.
Running inference on Cloudfleet
On Cloudfleet, an inference Pod that requests nvidia.com/gpu triggers node auto-provisioning of a matching GPU node, with NVIDIA drivers and the NVIDIA Container Toolkit installed, and labels such as cfke.io/accelerator-name select the GPU model. Cloud and on-premises GPU nodes can join the same cluster, as described in the GPU workloads documentation and on the GPU Kubernetes page.