How do you make Kubernetes highly available across clusters?
Highly available Kubernetes across clusters, regions or clouds is built in layers: a replicated control plane, nodes in several zones, applications with several replicas and disruption budgets, a second cluster in a separate failure domain, traffic steering that fails over between clusters, and data replicated to wherever the standby runs. Each layer removes one single point of failure. Running more than one cluster removes the control plane itself as a single point of failure, at the cost of keeping configuration, traffic and data in sync.
The layers of availability
| Layer | Failure it covers | Mechanism |
|---|---|---|
| Control plane | Loss of an API server or etcd member | Three or five etcd members across zones |
| Nodes | Loss of a node or a whole zone | Nodes in at least three zones, Pods spread across them |
| Application | Pod crashes, node drains, upgrades | Multiple replicas, PodDisruptionBudgets, readiness probes |
| Cluster | Control plane failure, failed upgrade, region outage | A second cluster in another region or provider |
| Traffic | A cluster or region becomes unreachable | DNS failover with health checks, anycast or a global load balancer |
| Data | Data loss, corruption or an unreachable primary | Replication, backups restorable in another environment |
The first three layers work inside one cluster. The last three exist for failures that take out a whole cluster or region.
Availability inside one cluster
Control plane. etcd needs a majority of its members to accept writes. Three members tolerate one failure, and five tolerate two. Kubernetes describes two layouts: stacked, where etcd runs on the control plane nodes, and external, where etcd runs on separate hosts (Options for Highly Available Topology). Place the members in different zones. The Kubernetes control plane article explains what stops when it is unavailable, and what a managed Kubernetes SLA covers explains what providers guarantee for it.
Nodes and zones. Use topology spread constraints on topology.kubernetes.io/zone so that replicas of one Deployment do not end up in the same zone.
Applications. A PodDisruptionBudget limits how many replicas voluntary disruptions, such as node drains during upgrades, may take down at once. Readiness probes keep traffic away from Pods that are not ready.
Active-active or active-passive clusters
With two or more clusters, you choose how they share the load.
| Active-active | Active-passive | |
|---|---|---|
| Traffic | All clusters serve users | One cluster serves, the other waits |
| Capacity cost | Each cluster carries part of the load, with headroom to absorb a failed peer | The standby either runs at full size or scales up during failover |
| Data | Needs multi-writer or carefully partitioned data | One primary, replicated to the standby |
| Recovery time | Seconds to minutes, limited by traffic steering | Minutes or longer, including promotion of the standby database |
| Complexity | Higher | Lower |
Active-passive is the common starting point, because most databases have one writable primary. Active-active fits stateless services and data stores built for multiple writers.
Steering traffic between clusters
User traffic usually fails over through DNS: health checks remove an unhealthy cluster’s address, and clients follow once the record’s TTL expires. Anycast addresses and global load balancers move traffic faster, because they do not depend on client DNS caches.
Traffic between services in different clusters needs its own discovery. The Multi-Cluster Services API exports a Service from one cluster with a ServiceExport resource and makes it reachable as <service>.<namespace>.svc.clusterset.local in the others. Its API version is still v1alpha1. Service meshes offer multi-cluster discovery and failover as well, described in what a service mesh is.
Data is the hard part
Stateless services fail over easily. Stateful ones decide your real recovery targets:
- Recovery point objective (RPO) is how much data you can afford to lose. Asynchronous replication between regions has an RPO above zero. Synchronous replication reaches zero, but adds the round-trip time between sites to every write.
- Recovery time objective (RTO) is how long recovery may take. It includes detecting the failure, promoting the standby and moving traffic.
Back up both data and cluster configuration to a location outside the failing environment, and restore into a different cluster or provider as a regular test. For self-managed control planes, etcd supports snapshot and restore for disaster recovery. Keep cluster configuration in Git so a replacement cluster can be rebuilt from it, as described in what GitOps is.
The trade-off
Multi-cluster availability removes the control plane as a single failure domain, but every cluster you add is another control plane to upgrade and another copy of configuration that can drift. A single cluster whose nodes span several clouds keeps one control plane and protects against the loss of a node provider, but not against the loss of the control plane. When nodes lose the control plane, running Pods keep running, as described in what happens when the control plane connection drops.
High availability with Cloudfleet
Cloudfleet Pro clusters replicate the control plane across availability zones with a 99.95% monthly uptime SLA, and Enterprise clusters carry a 99.99% SLA. Nodes in one cluster can run in several cloud providers and on-premises. See cluster types.