Many AI teams end up with two GPU estates: a training cluster full of 8-GPU servers, and a separate serving setup that runs models behind an API. Each has its own tooling, its own images, and its own idle capacity. Running both in one Kubernetes cluster removes that duplication. Training jobs get multi-GPU nodes with fast interconnects, inference services get smaller GPUs close to users, and GPU sharing can pack several small models onto one card instead of leaving most of it unused.
This post explains how training and inference differ, how to keep them apart inside one cluster, and when to share a GPU between workloads.
Training and inference need different GPUs
Training and inference both run on GPUs, but they stress them in opposite ways. Training is a long, throughput-bound batch job. A run keeps every GPU at full load for hours or weeks, exchanges gradients between GPUs constantly, and cares far more about interconnect bandwidth than about latency for any single request. Inference is a latency-bound service. Traffic arrives in bursts, a single model often fits in a fraction of one GPU’s memory, and the workload scales with users rather than with dataset size.
| Training | Inference | |
|---|---|---|
| Typical GPUs | H100, H200, B200, B300 | L4, L40S, RTX PRO 6000, or a slice of an H100 |
| GPUs per workload | 8 per node, often many nodes | A fraction of one GPU to a few GPUs |
| What limits it | GPU memory and interconnect bandwidth (NVLink, InfiniBand) | Latency and cost per request |
| Load pattern | Constant, near 100% for the length of the run | Bursty, follows user traffic |
| Interruptions | Tolerated with checkpoints, so spot capacity works | Not tolerated, needs spare replicas |
| Placement | Wherever the large GPUs are available and cheapest | Close to users and data |
These differences are the reason teams split the two. They are also the reason a single cluster works well, as long as the scheduler knows which nodes belong to which workload.
Why one cluster instead of two
A shared cluster gives training and inference the same container images, registries, secrets, observability, and deployment pipeline. A model trained on Monday is served on Tuesday by the same platform, with no export to another environment. Capacity moves too: when a training run ends, the nodes it used scale away, and the node auto-provisioner adds the inference capacity that traffic needs.
The risk of a shared cluster is interference. A training job that lands on an inference node takes all its GPUs, and an inference Pod on a training node blocks a whole 8-GPU server for one small model. Kubernetes solves this with standard tools: dedicated node groups, taints and tolerations, and priority classes.
The examples below use two priority classes. Create them once per cluster, because Kubernetes rejects Pods that name a priority class that does not exist:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: online-inference
value: 100000
description: "Inference services that serve user traffic"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-training
value: 1000
preemptionPolicy: Never
description: "Training jobs that can wait for capacity"
preemptionPolicy: Never keeps a training job from evicting other Pods while it waits, and the higher value lets inference preempt training where both share nodes.
Keeping training and inference apart
In Cloudfleet, a Fleet is a group of nodes with its own providers, constraints, and taints. A cluster for training and inference typically has two GPU Fleets next to the Fleet for general services. The training Fleet is constrained to large GPU instance families and tainted with workload=training:NoSchedule. The inference Fleet is constrained to inference GPUs and tainted with workload=inference:NoSchedule. Every node the Fleets provision carries the taint from the moment it joins, so nothing else lands there by accident.
A training job then requests whole GPUs and tolerates the training taint:
apiVersion: batch/v1
kind: Job
metadata:
name: finetune-llm
spec:
template:
spec:
restartPolicy: OnFailure
priorityClassName: batch-training
nodeSelector:
karpenter.sh/nodepool: training
cfke.io/accelerator-name: H100
tolerations:
- key: workload
operator: Equal
value: training
effect: NoSchedule
containers:
- name: trainer
image: registry.example.com/ml/trainer:2026-08
resources:
limits:
nvidia.com/gpu: 8
The inference service requests one GPU per replica, tolerates the inference taint, and runs with a higher priority, so it can preempt batch work in a shared Fleet if you choose to have one:
apiVersion: apps/v1
kind: Deployment
metadata:
name: chat-api
spec:
replicas: 3
selector:
matchLabels:
app: chat-api
template:
metadata:
labels:
app: chat-api
spec:
priorityClassName: online-inference
nodeSelector:
karpenter.sh/nodepool: inference
cfke.io/accelerator-name: L4
tolerations:
- key: workload
operator: Equal
value: inference
effect: NoSchedule
containers:
- name: server
image: registry.example.com/ml/chat-api:2026-08
resources:
limits:
nvidia.com/gpu: 1
When either workload is pending and no node fits, the node auto-provisioner adds a matching GPU node and installs the NVIDIA drivers and container toolkit on it. The cfke.io/accelerator-name, cfke.io/accelerator-memory, and cfke.io/provider labels let you pin a model or a cloud, and without them Cloudfleet picks the most cost-effective option that fits. The GPU workloads documentation lists the labels and supported models.
Sharing GPUs between inference workloads
A modern data center GPU has far more memory and compute than most inference services need. A 7B-parameter model quantized to 4 bits fits in a few gigabytes, and an embedding model or a reranker needs even less. Giving each of them a whole L40S or H100 leaves most of the card idle. GPU sharing lets several Pods use one physical GPU. There are three ways to do it on NVIDIA hardware.
Time slicing lets the GPU switch between processes from different containers, much like a CPU scheduler. Each Pod requests nvidia.com/gpu: 1 and gets a turn on the full GPU. Time slicing works on every NVIDIA GPU since the Pascal generation and needs no changes to the application. It does not limit how much memory each workload uses, so one Pod can still exhaust the card. It suits bursty services, development environments, and notebooks that sit idle most of the time.
Multi-Process Service (MPS) runs work from several CUDA processes concurrently on one GPU instead of switching between them. That removes the context switching overhead and lowers latency, which makes MPS a good fit for inference with small batch sizes. MPS in the NVIDIA device plugin also splits GPU memory and compute evenly between the Pods that share the card, so one service cannot starve the others.
Multi-Instance GPU (MIG) partitions an A100, H100, H200, or Blackwell GPU in hardware, into up to seven instances with their own memory, cache, and compute. MIG gives the strongest isolation of the three and predictable performance per slice, at the cost of fixed partition sizes.
| Time slicing | MPS | MIG | |
|---|---|---|---|
| How it shares | Takes turns on the whole GPU | Runs processes concurrently | Splits the GPU in hardware |
| Memory isolation | None | Even split between Pods | Full, per instance |
| GPUs supported | Pascal and newer | Volta and newer | A100, H100, H200, Blackwell data center GPUs |
| Best for | Bursty services, dev and test | Low-latency inference, many small models | Multi-tenant serving with strict isolation |
On Cloudfleet, time slicing and MPS are available on the Pro and Enterprise tiers. You choose the strategy and the maximum number of Pods per GPU in the cluster settings, and inference Pods keep requesting nvidia.com/gpu: 1 while sharing the physical card. The setting applies to every GPU node in the cluster, training nodes included, so an 8-GPU request on a shared cluster no longer guarantees eight dedicated cards. Support for MIG is coming. Teams that need dedicated training GPUs next to shared inference GPUs, or hardware isolation, can talk to us about it.
Training and inference capacity from different providers
Training and inference rarely want the same location. Large GPUs such as the B300 or H200 are often easiest to find at specialized GPU clouds, while inference belongs in the regions where your users and data are, or on servers you already own. Because one Cloudfleet cluster can span several clouds and on-premises servers, the training Fleet and the inference Fleet do not have to share a provider. A training run can use 8-GPU nodes at one provider while the inference Fleet runs L4 instances on AWS or Google Cloud, or GPU servers in your data center that join as self-managed nodes.
The GPU Kubernetes page lists current GPU offerings across more than 20 providers, and the Cloudfleet team helps with procurement when capacity is scarce.
Frequently asked questions
Can training and inference run in the same Kubernetes cluster?
Yes. Use separate node groups for each, such as Cloudfleet Fleets, taint them so only the right workloads land there, and give inference a higher priority class than batch training. Both then share images, pipelines, and observability, while each runs on the GPUs it needs.
Can multiple Pods share one GPU in Kubernetes?
Yes. With time slicing or MPS, the NVIDIA device plugin advertises each physical GPU as several schedulable GPUs, and each Pod requests nvidia.com/gpu: 1. With MIG, the GPU is split into hardware instances that Pods request individually.
What is the difference between GPU time slicing and MPS?
Time slicing lets processes take turns on the whole GPU and does not isolate memory. MPS runs processes concurrently and splits memory and compute between them, which gives lower latency and better isolation for inference.
Should training jobs use shared GPUs?
Usually not. Training keeps GPUs busy on its own and depends on full memory and interconnect bandwidth, so sharing slows it down. Sharing pays off for inference, notebooks, and development work that leaves GPUs idle.
How do I stop inference Pods from landing on training nodes?
Taint the training nodes and add the matching toleration only to training workloads. In Cloudfleet, Fleet taints apply to every node the Fleet provisions, including nodes the node auto-provisioner adds later.
