What is hybrid cloud Kubernetes, and how do you run it?
Hybrid cloud Kubernetes is the practice of running containerized workloads on your own infrastructure (data centers, colocation racks or edge sites) and on one or more public clouds, managed through the Kubernetes API. Teams build it in one of three ways: separate clusters per environment tied together by a fleet manager, one cluster whose control plane manages nodes in every environment, or a cloud provider’s Kubernetes service extended onto their own hardware.
What makes a setup hybrid
NIST defines a hybrid cloud as a composition of two or more distinct cloud infrastructures that “remain unique entities, but are bound together by standardized or proprietary technology that enables data and application portability” (NIST SP 800-145). Kubernetes is the portability layer in that definition. A Deployment that runs on a server in your rack also runs on a cloud virtual machine, as long as both run conformant Kubernetes.
Hybrid cloud is not the same as multi-cloud, which means using several public cloud providers. The two often combine. The comparison of multicloud and hybrid cloud covers the difference.
Three ways to run Kubernetes across on-premises and cloud
| Pattern | Control plane | Nodes | Main trade-off |
|---|---|---|---|
| Separate clusters plus a fleet manager | One per environment | Local to each cluster | Each site keeps working when links fail, but you operate many control planes and move workloads between clusters yourself |
| One cluster with nodes in several environments | One, usually managed, in one location | On-premises and cloud nodes join the same cluster | One scheduler and one API across all capacity, but every site depends on its link to the control plane |
| Cloud provider extension | In the provider’s region or on your hardware, depending on the product | Your servers | Consistent with the provider’s managed service, but ties hybrid operations to that provider |
Separate clusters per environment
Each data center and each cloud region gets its own cluster. A GitOps pipeline or a Kubernetes fleet management tool applies the same configuration, policies and add-ons to all of them. This pattern suits sites that must keep running while disconnected, such as factories or ships. The cost is operational: every cluster has its own control plane to upgrade and back up, and moving a workload from on-premises to cloud means redeploying it to a different cluster.
One cluster spanning environments
Kubernetes uses a hub-and-spoke pattern: all API usage from nodes terminates at the API server (Communication between Nodes and the Control Plane). A node in your data center can therefore register with a control plane hosted elsewhere, as long as it can reach the API server over a secure connection. Workloads then use labels, node affinity and taints to choose between on-premises and cloud nodes through one API. The trade-off is that nodes need an encrypted network between sites, and a site that loses its link stops receiving scheduling decisions until it reconnects.
Cloud provider extensions
The large cloud providers each offer a hybrid product:
- Amazon EKS Hybrid Nodes attaches on-premises servers to an EKS control plane in an AWS Region. AWS states that it needs a reliable connection to AWS and is not a fit for disconnected environments, and bills per vCPU-hour of attached nodes.
- Amazon EKS Anywhere runs the whole cluster, control plane included, on your hardware.
- Azure Arc-enabled Kubernetes attaches clusters running anywhere to Azure for configuration, policy and monitoring.
- Google Distributed Cloud runs Google’s Kubernetes software on your bare metal or VMware servers, or on Google-managed hardware in your facility, including air-gapped deployments.
For a closer look at the AWS options, see EKS Anywhere vs EKS Hybrid Nodes. The hybrid Kubernetes platform comparison covers pricing and trade-offs across vendors.
What every hybrid design has to solve
Connectivity. Nodes need a path to the control plane, and Pods need routable IP addresses between sites. Teams use private links, site-to-site VPNs or an encrypted overlay such as WireGuard®, and must avoid overlapping address ranges between data center and cloud networks. The mechanics are in how nodes in different clouds join one cluster network.
Identity and policy. Use one RBAC model, one set of admission policies and one way for workloads to reach cloud APIs in every environment. Otherwise the same manifest behaves differently on-premises and in the cloud.
Storage and data gravity. Persistent volumes are bound to a site, and storage classes differ between your SAN and a cloud block store. Keep stateful workloads next to their data and move stateless ones freely. The options are covered in storage for on-premises and hybrid Kubernetes.
Link failures. When a node loses its connection to the control plane, running Pods keep running, but nothing is rescheduled or rolled out until the link returns. The details are in what happens when the control plane connection drops.
Upgrades. The version skew policy lets a kubelet run up to three minor versions behind the API server, but never ahead of it. Upgrade the control plane first, then nodes in every environment, before the oldest falls out of the supported window.
Hybrid Kubernetes with Cloudfleet
Cloudfleet Kubernetes Engine (CFKE) uses the single-cluster pattern: a managed control plane with nodes that node auto-provisioning creates in AWS, GCP and Hetzner Cloud, plus any Linux server in your data center added as a self-managed node. All nodes connect over an encrypted WireGuard® overlay and need only outbound internet access.