Public cloud load balancing

This page describes Kubernetes Services and their use in Cloudfleet Kubernetes Engine (CFKE). Kubernetes has different types of Services, which group a set of Pod endpoints into a single resource. Before you read this page, read the Kubernetes documentation on Services. Also read the CFKE networking architecture to understand how nodes communicate across clouds and regions. To diagnose problems with existing load balancers, see Load balancer issues.

A Service aggregates a collection of Pod endpoints into one resource. You can set up multiple methods to access this aggregation. Typically, the Service has a stable cluster IP address that clients within the cluster use to reach its Pods. When a client sends a request to this stable IP address, the request goes to one of the Pods in the Service. There are five types of Services:

• ClusterIP (default)
• NodePort
• LoadBalancer
• ExternalName
• Headless

ClusterIP, ExternalName, and Headless Services work the same in CFKE as in any other Kubernetes cluster. The Kubernetes documentation gives a good overview of these types of Services.

NodePort

When you create a Service with the NodePort type, CFKE opens a static port on the cluster’s nodes. You can access the Service using the node’s public IP address and the port number. See the Kubernetes documentation for more information.

TCP load balancing

Cloudfleet can provision TCP load balancers in some of the cloud providers that node auto-provisioning supports. This feature provisions and manages TCP load balancers across different cloud providers and regions. It uses the standard implementation of Kubernetes Services.

The table below shows the current status of the TCP load balancing features:

When you create a Service with the LoadBalancer type, many cloud providers automatically provision their own load balancer. The load balancer routes traffic to the Service in the region where the cluster runs. This architecture works for classic cloud providers, because their clusters place compute nodes in only one region. CFKE is different: it can place compute nodes in different cloud providers and regions.

In CFKE, when you create a Service with the LoadBalancer type, CFKE by default provisions a load balancer in every cloud provider and region where at least one node exists. The Service shows the IP addresses of these load balancers in the LoadBalancer Ingress field, so expect multiple IP addresses there. With this setup, you serve traffic from all cloud providers and regions where the cluster is present.

Multiple load balancers cost extra, and routing through the in-cluster network can add network hops. To avoid both, you can set the External Traffic Policy to Local. To learn more, see the Load balancing in multi-cloud environment section.

Supported annotations

You can configure load balancing with annotations. CFKE supports these annotations on the Service object:

Example: Exposing a service using a TCP load balancer

Before you start, make sure you have a Deployment with Pods on auto-provisioned nodes from supported cloud providers.

To create an external load balancer, add the following line to your Service manifest:

   type: LoadBalancer

Your manifest might then look like:

apiVersion: v1
kind: Service
metadata:
 name: example-service
spec:
 selector:
   app: example
 ports:
   - port: 80
     targetPort: 80
 type: LoadBalancer
 externalTrafficPolicy: Local

Setting up DNS for your load balancer service

To find the IP address created for your Service, get the Service information with kubectl:

kubectl describe services example-service

The output is similar to:

Name:                     example-service
Namespace:                default
Labels:                   app=example
Annotations:              <none>
Selector:                 app=example
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.3.22.96
IPs:                      10.3.22.96
LoadBalancer Ingress:     142.132.244.132 (VIP), 2a01:4f8:c011:2c4::1 (VIP)
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  30593/TCP
Endpoints:                172.17.0.3:9376
Session Affinity:         None
External Traffic Policy:  Local
Events:                   <none>

The LoadBalancer Ingress field shows the IP addresses of the load balancer. In most cases, the field has multiple IP addresses, one for each cloud provider and region where at least one node exists. The IP addresses in LoadBalancer Ingress can also change when:

• The Pods behind the load balancer Service move to different regions or cloud providers
• The externalTrafficPolicy is set to Local and Pod placement changes

Cloud providers typically support one load balancer per region. If Cloudfleet moves the Pods to a different region or cloud provider to optimize resources, it creates a new load balancer there and removes the old one. For this reason, do not point your final DNS record directly at the IP addresses. Cloudfleet provides a stable DNS name for each load balancer Service that you can use in your DNS configuration. The DNS name has this format:

<service-name>.<namespace>.<cluster-id>.<control-plane-region>.cfke.cloudfleet.dev

For example, take a cluster in the europe-central-1a region with the ID 79f9275a-eec6-4dd9-af0a-78604eacb773. If the example-service Service is in the default Namespace, its DNS name is:

example-service.default.79f9275a-eec6-4dd9-af0a-78604eacb773.europe-central-1a.cfke.cloudfleet.dev

To reduce how often load balancers are recreated, you can use a node selector to keep the Pods behind a load balancer in one cloud provider and region. For example, to keep the Pods in the Hetzner Nürnberg region, use this node selector:

spec:
  template:
    spec:
      nodeSelector:
        cfke.io/provider: hetzner
        topology.kubernetes.io/region: nbg1
      # Your pod specification

With this configuration, CFKE creates the load balancer only in this region. The Pod can still reach other Pods in the cluster through the in-cluster network.

Using your own IP addresses (Hetzner)

On Hetzner, a load balancer can use Primary IPs that you own, so its address stays the same even when CFKE deletes and recreates the load balancer. This is useful when you need a fixed IP, for example for firewall allowlists.

1. In the Hetzner Cloud Console or with the Hetzner API, create an IPv4 Primary IP, an IPv6 Primary IP, or both, in the Hetzner project used by your Fleet. Create them in the location where the load balancer will run, for example nbg1. Keep auto delete disabled, which is the default. With auto delete enabled, Hetzner deletes the IP together with the load balancer.
2. Reference the Primary IPs on the Service with the networking.cfke.io/hetzner-primary-ips annotation. Use the name or the address of each Primary IP, at most one IPv4 and one IPv6. For IPv6, you can use the /64 network, for example 2a01:4f8:1c1f:7ae1::/64, or any address in it.
3. A Primary IP belongs to one location, so pin the Pods behind the Service to that location with a node selector.

apiVersion: v1
kind: Service
metadata:
  name: web
  annotations:
    networking.cfke.io/hetzner-primary-ips: "web-ipv4,web-ipv6"
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
  type: LoadBalancer
  externalTrafficPolicy: Local
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      nodeSelector:
        cfke.io/provider: hetzner
        topology.kubernetes.io/region: nbg1
      containers:
        - name: web
          image: nginx
          ports:
            - containerPort: 80

CFKE never creates or deletes your Primary IPs. When the load balancer is deleted, for example because no Pods of the Service run in its location anymore, the Primary IPs become unassigned. CFKE assigns them again when it recreates the load balancer.

Keep in mind:

• The Primary IPs must exist, be in the same location as the load balancer, be unassigned, and not be blocked. The load balancer must be external, not Internal. Otherwise, CFKE does not create the load balancer and reports a Warning event on the Service.
• The annotation applies when CFKE creates the load balancer. If you add it to a Service whose load balancer already exists, delete and recreate the Service.
• With externalTrafficPolicy: Cluster and nodes in several locations, CFKE creates a load balancer in each location, and only the one in the Primary IP’s location can use it. Use externalTrafficPolicy: Local together with the node selector.

Load balancing in multi-cloud environment

“LoadBalancer Ingress” contains the IPv4 and IPv6 addresses of the load balancers created for your Service.

Other Kubernetes solutions typically handle load balancing for only one region of one cloud provider. Cloudfleet supports multiple cloud providers and regions in one cluster. This difference affects how CFKE provisions load balancers.

Kubernetes LoadBalancer Services have a setting called “External Traffic Policy” with the value “Local” or “Cluster.”

• With “Cluster,” the Service is exposed on all nodes of the cluster. If a node has no Pod that serves the Service, the in-cluster network routes traffic to a serving Pod.
• With “Local,” the Service is exposed only on the nodes that have a Pod of the Service.

A Cloudfleet cluster can have nodes from different clouds and regions. When the External Traffic Policy is Cluster, Cloudfleet tries to expose the Service on all nodes in the cluster. Therefore, it provisions load balancers in every cloud provider and region where at least one node exists. The Service shows the IP addresses of these load balancers in the LoadBalancer Ingress field.

This mode gives maximum flexibility, and you can serve traffic from all cloud providers and regions where the cluster is present. It also has trade-offs:

• Multiple load balancers can cost extra
• If a node does not directly serve a Pod, traffic goes through the in-cluster network. This route adds network hops and can strain the in-cluster network, which is encrypted.

To avoid these risks, set the External Traffic Policy to Local. In this mode, Cloudfleet provisions load balancers only in cloud providers and regions where Pods of this Service run. The disadvantage: if the Pods move to another cloud, or to another region in the same cloud, CFKE creates new load balancers and the IP addresses change. In this case, you can use solutions like External DNS to keep the list of IP addresses up to date.

Ingress and Gateway API support

Services of type LoadBalancer work at L4 (TCP and UDP), so they are not aware of the HTTP/HTTPS protocol. For HTTP/HTTPS-aware load balancers, use Ingress or Gateway API resources, which are native Kubernetes APIs.

CFKE does not support Ingress and Gateway API resources out of the box, because customers have different preferences for these resources. You can use any Ingress Controller or Gateway API implementation.

NGINX Ingress Controller and Traefik are two of many solutions that the Cloudfleet team and the community have tested. For a guide that adds automatic HTTPS certificates with cert-manager, see the Install NGINX Ingress Controller and cert-manager tutorial.