How much GPU memory do you need to run an LLM?
An LLM needs enough GPU memory for its weights, its KV cache and the serving framework’s overhead. Weights take the number of parameters times the bytes per parameter: an 8-billion-parameter model needs about 16 GB at 16-bit precision, 8 GB at 8-bit and 4 GB at 4-bit. The KV cache grows with context length and the number of concurrent requests, and for a busy server it can be as large as the weights.
Weights take parameters times precision
| Precision | Bytes per parameter | 8B model | 70B model |
|---|---|---|---|
| FP16 or BF16 | 2 | 16 GB | 140 GB |
| FP8 or INT8 | 1 | 8 GB | 70 GB |
| INT4 or FP4 | 0.5 | 4 GB | 35 GB |
Quantized formats add a little for scaling factors, and some layers usually stay at higher precision, so real checkpoints come out slightly larger than the table.
Mixture-of-experts (MoE) models need memory for all experts, not only the active ones. Qwen3-30B-A3B activates 3.3 billion parameters per token but has 30.5 billion in total, so its BF16 weights take about 61 GB. OpenAI’s gpt-oss-120b has 117 billion parameters with 5.1 billion active, and fits on a single 80 GB GPU because its expert weights are stored in the 4-bit MXFP4 format. Active parameters set the compute cost per token. Total parameters set the memory.
The KV cache grows with every token in flight
During generation, the model stores a key and a value vector for every token of every request, in every layer. That is the KV cache. Its size per token is:
KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value
The 2 counts keys and values. Read the other numbers from the model’s config.json (num_hidden_layers, num_key_value_heads, and head_dim or hidden_size divided by num_attention_heads). Total KV cache is that figure times context length times concurrent requests.
Grouped-query attention (GQA) is the reason estimates online disagree. In a GQA model, several query heads share one KV head, so the cache shrinks by the ratio of query heads to KV heads. Llama 3.1 70B has 64 attention heads but only 8 KV heads, per Meta’s model parameters. Calculating with 64 heads gives 2.5 MiB per token. Calculating correctly with 8 gives 320 KiB, eight times less.
| Model | Layers | KV heads | Head dim | KV cache per token (FP16) |
|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB |
| Llama 3.1 70B | 80 | 8 | 128 | 320 KiB |
| Qwen3-30B-A3B | 48 | 4 | 128 | 96 KiB |
An 8-bit KV cache halves these numbers. vLLM sets it with --kv-cache-dtype.
Framework overhead
Serving frameworks need memory for activations, CUDA graphs and the CUDA context, and they reserve the rest for the KV cache up front. vLLM claims a fixed fraction of GPU memory set by --gpu-memory-utilization, which defaults to 0.92 in the current release documentation as of November 2026. Whatever remains after the weights and activations becomes KV cache, and vLLM logs how many tokens fit. Plan for a few gigabytes of overhead beyond weights and cache.
Three worked examples
Llama 3.1 8B, chat assistant. BF16 weights take 16 GB. Sixteen concurrent requests of 8,192 tokens hold 131,072 tokens, at 128 KiB each that is 16 GiB of KV cache. Around 35 GB in total fits a 48 GB L40S. With FP8 weights the model also fits a 24 GB L4, at lower concurrency.
Llama 3.1 70B, internal assistant. BF16 weights take 140 GB, more than any single H100. FP8 weights take 70 GB and fit one 141 GB H200, which leaves roughly 55 GB for the KV cache after overhead: about 170,000 tokens at 320 KiB each, enough for 20 requests of 8,192 tokens. Keeping BF16 means splitting the model over two or more GPUs.
Qwen3-30B-A3B, MoE. All 30.5 billion parameters stay in memory: 61 GB in BF16 or about 31 GB in FP8. At 96 KiB per token, the KV cache is small for the model’s size, so an FP8 copy runs well on a single 48 GB or 80 GB GPU.
Quantize or split across GPUs
When a model does not fit, there are two options. Quantization to FP8 or 4-bit cuts weight memory by half or more on the same GPU, at some cost in output quality that you should measure on your own prompts. Tensor parallelism splits every layer over several GPUs in one node. The vLLM guide recommends it when a model is too large for one GPU but fits on one multi-GPU node, and recommends pipeline parallelism across nodes or on GPUs without NVLink, such as the L40S. Splitting adds communication between GPUs, so on PCIe-only servers it scales worse than on NVLink systems.
Fine-tuning needs more
Training stores far more than inference. Full fine-tuning with the Adam optimizer in mixed precision needs about 16 bytes per parameter for weights, gradients and optimizer states, as the ZeRO paper breaks down, before counting activations. That is roughly 128 GB for an 8B model. LoRA freezes the base model and trains small adapter matrices, which removes most gradient and optimizer memory. QLoRA goes further by keeping the frozen base in 4-bit, and its authors fine-tuned a 65B model on a single 48 GB GPU.
Memory by GPU
| GPU | Memory per GPU |
|---|---|
| NVIDIA L4 | 24 GB |
| NVIDIA L40S | 48 GB |
| NVIDIA A100 | 40 GB or 80 GB |
| NVIDIA H100 | 80 GB (SXM), 94 GB (NVL) |
| NVIDIA RTX PRO 6000 Blackwell | 96 GB |
| NVIDIA H200 | 141 GB |
| NVIDIA B200 | 180 GB (1,440 GB per 8-GPU DGX B200) |
| NVIDIA B300 | Up to 288 GB (2.1 TB per 8-GPU HGX B300) |
Per-GPU figures for A100, RTX PRO 6000, B200 and B300 are from NVIDIA’s MIG supported GPUs list. Cloud providers list B300 at 268 to 288 GB per GPU depending on the system.
Memory decides whether a model fits. Whether it is worth running it on a given GPU is a cost question, covered in how to reduce GPU costs. Several small models that each use a fraction of a GPU can share one GPU, and serving the result at scale is covered in how to run LLM inference on Kubernetes.
Picking GPUs by memory on Cloudfleet
On Cloudfleet, a Pod can select a GPU by memory with the cfke.io/accelerator-memory label, or by model with cfke.io/accelerator-name, and node auto-provisioning creates a matching node in a connected cloud account. The GPU workloads documentation lists the labels and supported GPU models.