Networking architecture

Kubernetes networking can be complex, but its networking model simplifies it: each Pod gets its own IP address. Containers within a Pod share the same IP, so Pods in a cluster communicate without NAT. Pods reach each other directly by IP address, which makes networking within the cluster easier to manage.

This document explains how Cloudfleet Kubernetes Engine (CFKE) manages networking and security within a Kubernetes cluster. CFKE provides efficient communication between Pods and secure data transfer across your infrastructure.

We recommend reading the Kubernetes Networking Guide to understand the basics of Kubernetes networking before you read this document.

Node-to-node communication on CFKE

Kubernetes is designed to run on a flat network. All nodes in a cluster communicate with each other directly, without NAT. A flat network is a challenge when a Kubernetes cluster runs across multiple cloud providers or on-premises infrastructure. CFKE solves this problem with a secure, encrypted network between nodes that uses WireGuard®. WireGuard® is a modern VPN solution that encrypts all traffic between nodes, which protects your cluster data from unauthorized access.

When a new node joins a CFKE cluster, it uses WireGuard® to establish a secure connection with the other nodes and the CFKE control plane. After the tunnels are established, the nodes get a secondary IP address and behave as if they are on the same flat network.

To establish WireGuard® tunnels, CFKE uses a combination of public and private keys. The public keys encrypt the traffic, and the private keys decrypt it. Immediately before the node joins the cluster, it shares its public key and gets the information about the other nodes from the control plane. Later, it uses the public keys of the other nodes to establish tunnels with them.

To create a tunnel with the control plane, exchange keys, and finally create tunnels with other instances, a CFKE node needs egress Internet access. The node can be behind NAT and does not need to have a public IP address. CFKE is designed to overcome the most complicated NAT setups and can typically establish tunnels directly with other nodes. If a direct connection is not possible, CFKE connects the nodes through a relay server over TCP 443. Relayed traffic stays encrypted end to end with WireGuard®, so the relay only forwards ciphertext. A relayed connection has higher latency and lower throughput than a direct one. For the ports and destinations that nodes need, see Network requirements.

The WireGuard® tunnels typically run over the public Internet. However, if two nodes are already on the same flat network, CFKE detects this and establishes the connection over the private network. Examples are two nodes in the same network segment in a datacenter, or in the same VPC in a cloud provider. In this case, the nodes use the Internet connection only at the start, to fetch the metadata for the VPN connection. The nodes then communicate over the private network. This is a great benefit for classical datacenter setups, where the nodes share a network but have no default encryption between them. CFKE always encrypts the communication between nodes.

The WireGuard® tunnels are highly performant, but they consume some node resources. They are tested to handle up to 10 Gbps of traffic per tunnel.

Container network interface (CNI)

The Container Network Interface (CNI) is a standard for creating plugins that configure network interfaces for Linux containers. CNI focuses only on network connectivity for containers and makes sure that resources are removed when containers are deleted. Its simple specification makes it widely supported and easy to implement.

In CFKE, Cilium is the default CNI plugin. Cilium is an open-source networking and security solution designed for Kubernetes clusters. Cilium uses eBPF (Extended Berkeley Packet Filter) technology to provide efficient load balancing, fast data processing, and secure communication between workloads. Cilium is built for large-scale, dynamic cloud-native environments. It supports clusters with thousands of nodes and hundreds of thousands of Pods, which makes it ideal for high-performance environments.

Cilium also enhances security with identity-based policies that go beyond traditional IP-based ACLs. With this fine-grained control, you define who can communicate within the cluster. This reduces your attack surface and protects your cloud-native applications.

On CFKE, Cilium works on the overlay network created by WireGuard®. CFKE customers can use all of Cilium’s advanced features, such as network policies, load balancing, and service mesh capabilities. For the available security layers and policy types, see Network security and Network policies.

Cilium’s observability layer Hubble records network flows on every CFKE node. To inspect these flows through the Hubble UI or CLI, see Network observability with Hubble.

Frequently asked questions

Q: My on-premise nodes are already on a private network. Why do I need WireGuard® tunnels between them?

Our experience in regulated industries has taught us the importance of a zero-trust networking architecture. Even within a “private” network, assuming inherent trust can be a vulnerability. Cloudfleet has secure defaults, so encryption of data in transit is standard practice, regardless of the network.

Here’s why this is crucial, even for your on-premise nodes:

• Defense in Depth: Network segmentation alone isn’t enough. If a perimeter is breached, or an insider threat exists, unencrypted internal traffic is exposed. Encrypting traffic between nodes provides an essential additional layer of security.

• Evolving Threat Landscape: Today, many organizations don’t own their networking infrastructure end-to-end, even within their own data centers. Shared infrastructure can introduce unseen vulnerabilities. “Private” networks can still be susceptible to physical taps or misconfigurations that could expose data.

• Lightweight & Efficient: WireGuard® is a modern, high-performance VPN protocol. It’s lightweight and uses minimal system resources, so the security benefits come with negligible performance overhead. WireGuard® secures inter-node communication without impacting your applications.

With WireGuard® tunnels, you adopt a proactive security posture that aligns with modern best practices and protects your data from a wider range of potential threats.

Q: The public clouds I use have exorbitant egress bandwidth fees. How can I avoid high bandwidth costs when running workloads across different clouds?

We recommend that you design your deployments with careful consideration of how application placement affects bandwidth usage. First, when one Pod communicates with another, CFKE uses its locality load balancing feature to route traffic to the nearest target Pod where it can. To get the most from this feature, you can use Kubernetes affinity rules. With affinity rules, you can co-locate Pods of services that frequently communicate with each other. CFKE’s node auto-provisioning respects these affinity rules and creates compute nodes that fulfill these location requirements. This helps to minimize cross-cloud bandwidth consumption.