Network connectivity issues

CFKE connects nodes across clouds and regions with an encrypted overlay network powered by Cilium. This page covers the in-cluster networking problems we most often see in support tickets and how to resolve them.

One pod is unreachable from other nodes

Symptom: connections to exactly one pod time out when they cross nodes, while connections from pods on the same node work. Typical manifestations: an admission webhook fails with dial tcp SERVICE_IP:443: i/o timeout, a Redis or database pod stops accepting cross-node connections, or connections hang in SYN_RECV. Restarting or recreating the pod does not help, and no network policy drops are logged.

Cause: a rare known issue where the pod’s internal network identity collides with a reserved value used for cross-node routing. Because the identity is derived from the pod’s labels, restarts reproduce the same identity. We are tracking this issue and a permanent fix is on the way; until it ships, use the workaround below.

Fix: add any new label to the pod so it receives a different identity. Recovery takes seconds:

kubectl -n NAMESPACE label pod POD_NAME cfke.io/identity-rehash=1

For a Deployment or other controller, add the label to the pod template instead, and persist it in your Helm values or GitOps configuration so a sync does not remove it:

kubectl -n NAMESPACE patch deployment DEPLOYMENT_NAME --type=strategic \
  --patch='{"spec":{"template":{"metadata":{"labels":{"cfke.io/identity-rehash":"1"}}}}}'

If you hit this repeatedly, contact support so we can confirm the diagnosis on our side.

Same symptom, different scope: if every newly created or rescheduled pod becomes unreachable across nodes (rather than one specific pod), while pods that existed earlier keep working, the cause is not an identity collision but a propagation problem in the cluster networking control plane, and the label workaround will not help. This is a platform-side condition: contact support with your cluster ID.

Network policies block access to the Kubernetes API

Symptom: a pod with a CiliumNetworkPolicy that allows egress to toEntities: [kube-apiserver] still fails to reach the API server with dial tcp 10.96.0.1:443: i/o timeout.

Cause: CFKE runs Cilium in kube-proxy replacement mode. The kube-apiserver entity matches the API server’s real endpoints, not the in-cluster Service ClusterIP (10.96.0.1) that client libraries use by default.

Fix: allow the ClusterIP explicitly alongside the entity rule. The ClusterIP is stable for the lifetime of the cluster:

egress:
  - toEntities:
      - kube-apiserver
  - toCIDR:
      - 10.96.0.1/32
    toPorts:
      - ports:
          - port: "443"
            protocol: TCP

Related pitfall: when testing network policies with a throwaway pod (kubectl run busybox ...), remember that label-based policies apply to the test pod too. A policy that only allows traffic between correctly labelled pods silently drops probes from an unlabelled debug pod, which is easy to misread as a platform fault. Test with pods that carry the labels your policies select on.

Tailscale and other tools that manage node networking

Symptom: the Tailscale Kubernetes operator (and similar tools that steer traffic with their own netfilter rules inside pods) cannot intercept connections; exposed services get no reply.

Cause: Cilium’s socket-level load balancing resolves Service addresses at connect time in the pod’s network namespace, before such tools see the traffic.

Fix: enable the “socket load balancing in host namespace only” option (socketLB.hostNamespaceOnly) for your cluster, available self-service under Edit cluster in the Cloudfleet Console. Applying it restarts the cluster networking components, which causes a brief traffic blip.

Also note that nested VPN setups (running a mesh VPN inside pods on top of CFKE’s own encrypted overlay) reduce the effective MTU and add a userspace hop, which can significantly limit throughput. Prefer exposing services through load balancers or egress gateways where possible.

All pod egress on one node is broken

Symptom: every pod on a single node loses internet access (curl https://1.1.1.1 times out from pods, works from the host); DNS lookups fail with server misbehaving; the same workloads on other nodes are fine.

Cause seen in practice: a workload or operator added an extra IP address (for example a provider floating IP) to the node’s network interface with ip addr add. The overlay network then selects that address as the source for pod egress, but since the provider never routed the address to this machine, all return traffic is blackholed.

Fix: remove the manually added address and stop the workload that adds it. If you need a fixed egress IP, do not attach floating IPs to auto-provisioned nodes. Instead, use a self-managed node with the floating IP properly assigned through the provider, pin the egress workloads to it, and configure an egress gateway.

Pods lose API server access after enabling an egress gateway

Symptom: after applying a CiliumEgressGatewayPolicy with destinationCIDRs: ["0.0.0.0/0"] to route all traffic through an egress gateway, the selected pods can no longer reach the Kubernetes API server.

Cause: 0.0.0.0/0 also matches traffic to the API server’s endpoint, which then gets routed out through the gateway instead of to the control plane.

Fix: exclude the API server endpoint from the policy. Its IP is cluster-specific; read it from the endpoints of the kubernetes Service:

kubectl get endpoints kubernetes -n default

Add the address to excludedCIDRs:

spec:
  destinationCIDRs:
    - "0.0.0.0/0"
  excludedCIDRs:
    - "API_SERVER_ENDPOINT_IP/32"

The endpoint IP can change during control plane maintenance; if pods lose API access again, re-check it and update the policy. If you are unsure about the right configuration, contact support. See Egress gateways for the full feature documentation.

Reaching private networks and registries

• Workloads need to reach on-premises services (internal DNS, databases, Active Directory): join a self-managed node that sits inside the private network and route selected pods through it with an egress gateway policy.
• Container images must come from publicly resolvable registries. A registry hostname that only resolves on your private VPN fails on CFKE nodes with ImagePullBackOff and no such host. Use a publicly reachable registry endpoint or the Cloudfleet Container Registry.