How do you isolate teams or customers in a shared Kubernetes cluster?

Kubernetes multi-tenancy means several teams or customers share one cluster, and you isolate them with a stack of controls: a namespace per tenant with RBAC, resource quotas, network policies and Pod Security, then dedicated nodes and sandboxed runtimes where tenants do not trust each other. Kubernetes has no built-in tenant object, so isolation is something you assemble. When tenants need their own cluster-wide resources or a hard boundary at the API, give each one a virtual control plane or a separate cluster.

Soft and hard multi-tenancy

The Kubernetes multi-tenancy guide describes two common cases. In multi-team tenancy, internal teams share a cluster, have some level of trust in each other and usually have kubectl or GitOps access. In multi-customer tenancy, a SaaS vendor or ISV runs workloads for customers who have no cluster access but must be strongly isolated from each other.

Soft multi-tenancy assumes tenants are not malicious and protects mainly against mistakes and noisy neighbors. Hard multi-tenancy assumes a tenant may be hostile and must not be able to read, affect or escape to another tenant’s workloads. The guide calls this “a spectrum” rather than a binary, and points out that the data plane, meaning the nodes, kernels and network, has a larger attack surface than the API.

The isolation ladder

Each rung adds isolation and cost. Most clusters combine several.

Namespace-level controls

A namespace per tenant, or per tenant workload, is the base layer. On its own a namespace isolates nothing at runtime. These controls make it a boundary:

• RBAC: bind each tenant’s group to Roles in its own namespaces only. Do not grant list on Secrets, Pod creation in other namespaces, or any cluster-scoped write access. See Kubernetes RBAC for the escalation paths to avoid.
• ResourceQuota and LimitRange: a ResourceQuota caps the total CPU, memory, storage and object count in a namespace, and a LimitRange sets per-container defaults and bounds, so one tenant cannot starve the others.
• Network policies: Pods in different namespaces can reach each other by default. A default-deny policy per namespace, plus explicit allow rules, turns that around. Kubernetes network policies covers the patterns and the DNS gotcha.
• Pod Security Admission: label tenant namespaces with the restricted or at least baseline level, so tenants cannot run privileged Pods, use host namespaces or mount hostPath volumes to reach the node.
• Storage: use dynamic provisioning with a Delete reclaim policy, so a released PersistentVolume is not handed to another namespace with data still on it.

The limit of this level is everything that is not namespaced. CustomResourceDefinitions, StorageClasses, admission webhooks and the API server itself are shared by all tenants, and quotas do not cover network bandwidth.

Node isolation and sandboxing

Dedicated nodes stop tenants from sharing a kernel. Taint each tenant’s node pool, and give that tenant’s Pods the matching toleration plus a node selector or affinity. In a multi-customer setup, a mutating admission policy usually adds these, so tenants cannot leave their pool by editing their own manifests. The upstream guide notes that the kubelet and the API server remain shared services, and that per-node charge-back becomes simpler.

When dedicated nodes per tenant cost too much, or one tenant runs untrusted code, a sandboxed runtime isolates each Pod’s kernel instead. Kubernetes’ guide calls containers “a weaker isolation boundary than virtual machines”. How to run untrusted code in Kubernetes compares gVisor and Kata Containers.

Virtual control planes and cluster per tenant

A virtual control plane gives each tenant its own API server, controller manager and data store, running as workloads inside a host cluster and syncing Pods to the shared nodes. Open-source projects such as vCluster take this approach. Tenants can install their own CRDs and operators and act as cluster administrators without touching each other. The upstream guide states the caveat: “per-tenant control planes do not solve isolation problems in the data plane”, so combine them with dedicated nodes or sandboxing for hostile tenants.

A cluster per tenant gives the strongest isolation and the simplest mental model. Each cluster has its own control plane, nodes and network, and an incident in one cannot reach another. The cost is operational: N control planes to pay for, upgrade, monitor and back up, plus per-cluster add-ons. Most of that cost is operational work rather than compute, which is what hosted control planes and fleet-level automation reduce.

How to choose

• Internal teams who trust each other: namespaces with RBAC, quotas, network policies and Pod Security are usually enough.
• Teams with different compliance scopes: add dedicated node pools, or separate clusters for the regulated scope.
• Customers running your software: a namespace or cluster per customer, with dedicated nodes or sandboxed runtimes if customers can supply code.
• Customers who need admin rights or their own CRDs: virtual control planes or a cluster per customer.

Multi-tenancy on Cloudfleet

On Cloudfleet Kubernetes Engine (CFKE), every cluster has its own control plane and data store that are never shared with other clusters, so a cluster per tenant gets an isolated API and etcd by default, as described in Security and data protection. Inside a shared cluster, Kata Containers is available as a managed runtime for tenants that must not share a kernel.