Which Kubernetes workloads can run on spot instances?
Kubernetes workloads can run on spot instances when losing a node with 30 seconds to two minutes of notice costs nothing more than a restart elsewhere. Stateless services with several replicas, batch jobs that checkpoint, and CI runners fit well. Singleton databases, quorum-based systems and long jobs that cannot resume do not, because a reclaim ignores PodDisruptionBudgets and can take several replicas at once.
Spot instances themselves, and how their pricing and notice periods differ between clouds, are covered in what spot instances are. This article covers what happens inside the cluster.
Which workloads fit
| Workload | Fit | Why |
|---|---|---|
| Stateless web and API services with 3 or more replicas | Good | Other replicas serve traffic while a lost Pod restarts on another node |
| Batch and data processing jobs that checkpoint | Good | A reclaimed job resumes from its last checkpoint instead of starting over |
| CI/CD runners | Good | Jobs are short and a failed job can be retried |
| Dev, test and preview environments | Good | Brief outages are acceptable |
| Inference replicas behind a load balancer | Good, with spare replicas | Model load time makes each restart slower, so keep extra capacity |
| Stateful sets with replicated storage and fast failover | Possible | Works if the system tolerates losing a member and rebuilding it |
| Singleton databases | Poor | Every reclaim is an outage and a risk to unflushed data |
| Quorum systems such as etcd, ZooKeeper or Kafka controllers | Poor | Losing two members at once can break quorum |
| Long training or rendering jobs without checkpoints | Poor | A reclaim throws away all progress |
The AWS Spot best practices draw the same line: spot works for stateless, fault-tolerant and flexible applications, and not for workloads that are stateful, fault-intolerant or tightly coupled between instances.
How a reclaim reaches Kubernetes
The cloud provider sends a notice, then shuts the instance down. Kubernetes only reacts if something on the node turns that notice into Pod terminations:
- Notice length. AWS gives two minutes. Azure sends a best-effort notice of up to 30 seconds. On GKE, a reclaimed Spot node gets a 30-second graceful shutdown by default: 15 seconds for regular Pods and 15 for critical system Pods.
- Graceful node shutdown. The kubelet’s graceful node shutdown feature delays the OS shutdown and terminates Pods first. It is enabled by default, but its grace periods default to zero, which turns it off until the node configuration sets
shutdownGracePeriod. - Termination handlers. Many setups run a handler that watches the provider’s metadata endpoint, then cordons and drains the node as soon as the notice arrives, so replacement Pods start before the instance disappears.
A Pod’s terminationGracePeriodSeconds cannot extend the notice. If your shutdown hook needs 60 seconds and the node gets 15, the process is killed mid-shutdown. Keep shutdown work shorter than the notice on your provider.
PodDisruptionBudgets do not stop a reclaim
A PodDisruptionBudget limits voluntary disruptions, such as drains and consolidation. A spot reclaim is involuntary, and the GKE documentation states that PDBs are not respected during preemption. If several replicas share one spot node, or several spot nodes of the same instance type in the same zone are reclaimed together, the budget does not protect you. Spread replicas instead.
Patterns that make spot safe
- On-demand base, spot burst. Run the minimum replica count that must survive on on-demand nodes, and let the autoscaler add the rest on spot. Node affinity on the provider’s capacity-type label, or topology spread constraints over it, keeps the split.
- Spread across instance types and zones. Spot capacity is priced and reclaimed per pool, meaning one instance type in one zone. AWS suggests being flexible across at least 10 instance types per workload. Topology spread constraints over
kubernetes.io/hostnameandtopology.kubernetes.io/zonekeep replicas from sharing a fate. - Taint spot nodes. A
NoScheduletaint on spot nodes keeps workloads off them unless they carry a matching toleration, so only workloads that were reviewed for interruptions land there. - Fall back to on-demand. When spot capacity runs dry, pending Pods need somewhere to go. Configure the node autoscaler to try on-demand when no spot pool can satisfy the request. AWS cautions that a mass failover can make on-demand capacity for the same instance type and zone harder to get, which is one more reason to stay flexible on instance types.
Spot GPUs
GPU spot capacity can save the most per hour, but it is reclaimed often and its availability changes by region. Cast AI’s 2026 report found fewer than 2% of GPUs in its dataset ran on spot through most of 2025. Training jobs need frequent checkpoints to durable storage, and inference replicas need spare capacity to cover model load time after a reclaim. How to reduce GPU costs covers the other options.
Spot is one lever among several; how to reduce Kubernetes costs ranks them, and running GitHub Actions runners on Kubernetes is a common first spot workload.
How Cloudfleet handles spot
In Cloudfleet Kubernetes Engine (CFKE), Fleet constraints allow or restrict spot and on-demand capacity per Fleet, and Pods select a capacity type with node affinity. Node auto-provisioning picks the cheapest matching instance type and replaces nodes that the cloud provider removes through spot preemption.