How do nodes in different clouds join one Kubernetes cluster network?

Nodes in different clouds join one Kubernetes cluster network through an encrypted overlay: each node builds tunnels to the other nodes over the internet, usually with WireGuard® or IPsec, and the CNI plugin routes Pod traffic through those tunnels. This meets the Kubernetes network model, which requires every Pod to reach every other Pod without NAT, something no single provider’s network can offer across providers. The engineering problems are NAT traversal, MTU overhead, the number of tunnels and cross-cloud latency.

What the Kubernetes network model requires

The Kubernetes networking model gives every Pod a unique cluster-wide IP and requires that all Pods can communicate with all other Pods, on the same node or on different nodes, without proxies or address translation. Inside one cloud, the provider’s virtual network routes between nodes and the CNI plugin handles the rest.

Across providers, that breaks. An AWS VPC and a Google Cloud VPC are separate private address spaces with no shared routing. Nodes often have no public IP and reach the internet only through a NAT gateway. On-premises nodes sit behind corporate firewalls. Something has to give every node a direct, routable path to every other node, and the Pod and node address ranges must not overlap anywhere.

Ways to connect nodes across providers

Kilo describes itself as a multi-cloud network overlay built on WireGuard® for Kubernetes. It groups nodes by location, such as a cloud provider or data center, and meshes the locations. Talos KubeSpan builds a full WireGuard® mesh between nodes and uses a discovery service so nodes can find each other.

CNI encryption alone does not solve reachability. Cilium’s WireGuard® mode, for example, needs UDP port 51871 open between all nodes, which assumes every node can already reach every other node’s address.

The multi-cluster approach keeps each provider’s network separate and connects the clusters instead. Submariner connects Pods and Services across clusters, and Cilium ClusterMesh does the same between Cilium clusters. Whether one cluster across clouds or one cluster per cloud is the better design is a separate decision; see whether one Kubernetes cluster can span multiple clouds.

NAT traversal

WireGuard® runs over UDP and needs at least one side of each tunnel to accept incoming packets. When both nodes sit behind NAT, the mesh needs help:

• Endpoint discovery. A coordination service tells each node the public address and port that its peers appear from. KubeSpan’s discovery service works this way, and its documentation recommends allowing inbound UDP on at least one end of each link.
• Keepalives. NAT devices drop idle mappings. WireGuard®’s PersistentKeepalive option sends a packet at a fixed interval to keep them open; the WireGuard® quick start suggests 25 seconds.
• Relays. When no direct path exists, some implementations forward the encrypted packets through a relay, often over TCP port 443, which passes most corporate firewalls. The relay sees only ciphertext, but the extra hop adds latency and caps throughput.

MTU overhead

Every tunnel adds headers, so the inner packet must be smaller than the network’s maximum transmission unit (MTU). Calico’s MTU guide lists 60 bytes of WireGuard® overhead for IPv4 and 80 bytes for IPv6. On a 1,500-byte network that leaves 1,440 or 1,420 bytes. Google Cloud VPC networks default to 1,460 bytes, so the tunnel MTU there drops to 1,400 for IPv4.

Use the lowest MTU of any path in the cluster, because a single node with a smaller MTU affects every connection to it. If the CNI also encapsulates with VXLAN or Geneve inside the tunnel, subtract both overheads. A wrong MTU shows up as connections that open fine and then hang on large responses or TLS handshakes.

How many tunnels a full mesh needs

A full mesh of n nodes needs n × (n − 1) / 2 tunnels. Ten nodes need 45; 100 nodes need 4,950, and each node tracks 99 peers. WireGuard® handles many peers per interface, but every node join, leave or address change has to reach every other node quickly. That is why mesh implementations distribute peer keys and endpoints from a central service instead of static configuration, and why some group nodes by location.

Latency, cost and scheduling

A cluster network makes every node reachable, but it does not make distant nodes close. Round trips between providers, or between a cloud region and a data center, take far longer than calls inside one zone, and most providers bill traffic that leaves their network as internet egress. See what cloud egress fees are.

Keep chatty workloads together. Use the topology.kubernetes.io/region and topology.kubernetes.io/zone node labels with affinity rules to place a service next to its database, and topology-aware routing to prefer nearby endpoints. Reserve cross-cloud paths for failover, burst capacity and workloads that tolerate the latency, such as batch jobs.

The control plane link

Nodes also need a reliable path to the control plane. The kubelet connects to the API server over HTTPS, and the API server connects back to kubelets for logs, exec and port forwarding. The Kubernetes documentation warns that by default the API server does not verify the kubelet’s serving certificate, which makes those connections unsafe over public networks unless they run through a tunnel. If the link drops, running Pods keep running but nothing new is scheduled; the post on what happens when the control plane connection drops covers the details. For what the tunnels do and do not encrypt, see whether Pod traffic is encrypted.

How Cloudfleet connects nodes

Cloudfleet Kubernetes Engine (CFKE) connects every node, in any cloud or data center, to the other nodes and the control plane through WireGuard® tunnels, with keys distributed by the control plane. Nodes need only outbound internet access and can sit behind NAT; when no direct path exists, traffic goes through an end-to-end encrypted relay over TCP 443. The networking architecture page explains the design.