On-premises load balancing with BGP

You have several options to expose your Kubernetes Service to external clients. Two common methods are NodePort and LoadBalancer. NodePort exposes a Kubernetes Service on each node’s IP at a static port. The LoadBalancer Service type is a Kubernetes abstraction that creates a network Load Balancer service. That Load Balancer exposes your Service through a single floating IP (VIP).

Public cloud providers typically offer managed Load Balancer services. They automatically provision a Load Balancer when you create a Service of type LoadBalancer. On-premises clusters have no such load balancing service to implement the Kubernetes LoadBalancer Service type. In this case, you need a third-party component to implement this functionality. If you do not run on a supported IaaS platform (GCP, AWS, Azure…), new LoadBalancers stay in the “pending” state indefinitely.

Bare-metal cluster operators then have only two lesser tools to bring user traffic into their clusters: “NodePort” and “externalIPs” Services. Both options have significant downsides for production use. As a result, bare-metal clusters are second-class citizens in the Kubernetes ecosystem.

One popular solution is the MetalLB project. MetalLB offers a network load balancer implementation that integrates with standard network equipment. Its goal is that external Services on bare-metal clusters also “just work” as much as possible. MetalLB can run in Layer 2 (ARP) mode or BGP mode.

In Layer 2 mode, MetalLB responds to ARP requests for the Service IP and sends traffic to the backend Pods with standard routing. In BGP mode, MetalLB establishes BGP peering with the network equipment and advertises the Service IP to it. The network equipment then routes traffic to the backend Pods.

Layer 2 mode is simpler to set up, but it has its own limitations and drawbacks. For example, this mode routes traffic to only one node at a time, which can cause suboptimal routing and performance.

BGP mode is more complex to set up, but it offers more flexibility and scalability. In BGP mode, all nodes that have a Pod for the Service announce the Service IP. The network equipment can then route traffic to multiple nodes at the same time. BGP mode can give better performance and more efficient use of resources.

If you are experienced with MetalLB, you can install it on your CFKE cluster and it works as expected. If you plan to use MetalLB in BGP mode and have no specific reason to use MetalLB, use Cloudfleet’s Cilium-based BGP advertisement feature instead. We recommend this feature because Cloudfleet officially supports it.

Cloudfleet uses Cilium as the Container Network Interface (CNI) plugin for Kubernetes. (See Network Architecture) Cilium supports BGP peering with the network equipment out of the box. This feature is a direct replacement for MetalLB’s BGP mode. You can also configure Cilium to advertise Pod IPs to the router, which exposes the Pod network directly to the external network. This can be useful when external network clients need to connect to the Pod network directly.

BGP announcement configuration

This section is a brief guide to configure BGP announcement in Cloudfleet. For full details on how the BGP announcement feature works, see the Cilium documentation.

First, create an IP pool from which Cilium selects LoadBalancer IPs. Use the following manifest as an example:

yaml

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: "example-pool"
spec:
  blocks:
    - start: 172.16.90.2 # Start of the IP range
      stop: 172.16.90.254 # End of the IP range

    - cidr: 172.16.91.0/24 # Alternative notation with CIDR

Next, configure the BGP advertisement for the LoadBalancer Services and, optionally, for the Pod network:

yaml

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: services
  labels:
    advertise: bgp # This label is used in the next step
spec:
  advertisements:
    - advertisementType: PodCIDR # This is optional and can be omitted if you don't want to advertise the Pod network

    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchExpressions:
          # To enable BGP advertisement for all LoadBalancer services, you can use the following expression
          # See https://docs.cilium.io/en/latest/network/bgp-control-plane/bgp-control-plane-v2/#multipool-ipam to learn why
          - { key: somekey, operator: NotIn, values: [ 'never-used-value' ] }

This example advertises all LoadBalancer Services. You can customize the configuration to advertise only specific Services. The LoadBalancerIP is the IP address selected from the IP pool you created earlier. The selector field selects the Services to advertise.

Next, configure the BGP-specific settings for this advertisement. Use the following manifest as an example:

yaml

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
    name: tor-rack-1
spec:
    gracefulRestart:
        enabled: true
        restartTimeSeconds: 15
    families:
        - afi: ipv4
          safi: unicast
          advertisements:
              matchLabels:
                  advertise: "bgp"

See the CiliumBGPPeerConfig documentation for details about the configuration options. The matchLabels field selects the advertisements to use for the BGP announcement. In this example, the advertise: bgp label selects the advertisement you created in the previous step.

Next, configure a BGP announcement for each router in your network. Use the following manifest as an example:

yaml

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: tor-rack-1
spec:
  nodeSelector:
    matchLabels:
      topology.kubernetes.io/region: berlin
      topology.kubernetes.io/zone: rack-1
  bgpInstances:
    - name: "cloudfleet"
      localASN: 65001 # The ASN number you want to use for your cluster
      peers:
        - name: unifi
          peerASN: 65000 # Router's ASN number
          peerAddress: "172.16.10.1" # Router's IP address
          peerConfigRef:
            name: tor-rack-1 # Reference to the peer configuration. We created this in the previous step

In this example, the nodeSelector field selects the nodes that are part of the BGP cluster. This setting is particularly important in Cloudfleet, because Cloudfleet allows nodes in different datacenters and clouds. The example configures the advertisement for the nodes in the berlin region and rack-1 zone, where the router is also located. If you have multiple datacenters, use different labels to create completely different BGP configurations for them.

The bgpInstances field configures the BGP instances for the nodes in the cluster. The peers field configures the BGP peers for the BGP instances. The peerConfigRef field references the BGP peer configuration you created in the previous step.

After you apply all the resources to your cluster and configure the router, check the BGP configuration with the Cilium CLI. To see the peer connections, use:

bash

$ cilium bgp peers

To see the advertisements, use:

bash

$ cilium bgp routes

Also confirm that the advertisements propagate correctly on the router side.

When everything is configured correctly, create a LoadBalancer Service in your cluster. Cilium assigns the Service an IP address from the IP pool you created earlier. Cilium advertises this IP address to the router, and the router routes the traffic to the correct node in the cluster.

On-premises load balancing with BGP

BGP announcement configuration

Further reading