Self-managed node issues

Self-managed nodes let you join any Linux server to a CFKE cluster. Unlike auto-provisioned nodes, you own the machine, its operating system, and its network. This page covers the failures we most often see, and where the responsibility boundary between you and Cloudfleet runs. For setup, see Self-managed nodes.

The join command fails

cloudfleet clusters add-self-managed-node connects to the machine over SSH and prepares it step by step. The two most common failures:

Package repository setup fails. If the “Setup Required Repositories” phase fails while downloading from pkgs.k8s.io, test IPv4 and IPv6 reachability separately from the node:

curl -4 -sS -o /dev/null -w '%{http_code}\n' https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key
curl -6 -sS -o /dev/null -w '%{http_code}\n' https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key

A 403 only on IPv6 means your provider’s IPv6 range is blocked by the CDN serving the Kubernetes packages, and the installer’s downloads prefer IPv6. This is a recurring problem on some VPS providers. Prefer a machine with a routed public IPv4 address; an IPv6-only or unrouted-IPv4 server also breaks image pulls from IPv4-only registries (such as ghcr.io) and pod egress later, with symptoms like dial tcp IP:443: i/o timeout even though kubectl still works.

SSH authentication fails. The CLI expects a private key file; agent-only setups (for example 1Password’s SSH agent) may not work. Instead of the SSH flow, generate the node’s join configuration as cloud-init data with the Terraform provider and apply it yourself:

data "cloudfleet_cfke_node_join_information" "node" {
  cluster_id = CLUSTER_ID
}

See the Terraform documentation and the provider-specific guides for examples.

The node freezes when workloads start

Symptom: the machine becomes unreachable over SSH whenever kubelet and the container runtime start, or shortly after a specific workload is scheduled.

Cause seen in practice: a workload without CPU limits (for example CPU-based LLM inference) consumed every core, starving kubelet, the container runtime, and SSH itself. The node appears dead although the hardware is fine.

Fix: set CPU limits on heavy workloads so cores remain for the system (for example, cap a workload at 12 of 16 cores), and set realistic requests everywhere.

This failure mode is not specific to self-managed nodes: a workload without resource requests and limits can starve the system components on any node and destabilize the whole cluster. Missing resource requests are the single most common root cause behind unstable clusters we see in support. On self-managed nodes the impact is worse because no provider replaces a starved machine automatically, and on auto-provisioned nodes missing requests additionally break node sizing (see Pods stuck in Pending).

Pods stop starting, existing pods go Unknown

Symptom: the node reports healthy, but no new container starts and existing pods drift into Unknown.

Cause seen in practice: /etc/resolv.conf disappeared on the host (systemd-resolved stopped or the symlink was removed). The container runtime requires it to create every pod sandbox.

Fix: restore DNS resolution on the host, then re-add the node:

sudo systemctl restart systemd-resolved
# or recreate the symlink:
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

The node is unreachable and logs fail with “No agent available”

Symptom: the node is NotReady, kubectl logs for pods on it fails with No agent available, and SSH times out.

Cause: the machine itself is down or cut off, often a provider incident on the physical host.

What to know: Cloudfleet cannot see, escalate, or resolve incidents in your provider account; check the provider’s status page and open a ticket with them. This is the key operational difference from auto-provisioned nodes, which Cloudfleet replaces automatically when a host fails.

Do not install a cloud provider integration

Do not install a cloud controller manager (for example the Hetzner cloud-controller-manager) or a provider node agent on CFKE nodes. CFKE ships its own cloud integration that manages load balancers (including PROXY protocol support) with its own annotations; a second controller conflicts with it and has caused cluster-wide networking outages. For persistent volumes, follow the provider-specific CSI guidance instead (see Persistent volume issues).

Also leave the host firewall permissive for the cluster’s overlay traffic; custom firewall rules that block it cut pods off from the API server (dial tcp 10.96.0.1:443: connect: no route to host from pod-network pods while host-network pods still work).

Upgrades

Self-managed nodes are synchronized with the control plane version by re-adding them. See Kubernetes versions and upgrades.