Self-managed nodes

For supported cloud providers (AWS, GCP, and Hetzner Cloud), Cloudfleet provides node auto-provisioning. Node auto-provisioning creates and manages compute nodes based on your workload requirements. For on-premises infrastructure and other cloud providers, you provision compute nodes manually and register them with Cloudfleet. This guide explains how to add self-managed nodes to your CFKE cluster.

Not sure which approach to use? See How Cloudfleet provisions nodes for a comparison of node auto-provisioning versus self-managed nodes.

Having problems with an existing node? See Self-managed node issues for join failures, frozen nodes, and other common problems.

Self-managed nodes are Linux machines that you provision and give to CFKE to run workloads. With them, you can expand your cluster with your own hardware or third-party cloud resources. Self-managed nodes are equal members of the cluster and can run any workload that their hardware supports.

Bare-metal self-managed nodes are also a good fit for Kata Containers. Kata Containers run untrusted workloads inside lightweight VMs and need hardware virtualization on the node.

To use the disks of self-managed nodes as persistent volumes, see Local disks.

Requirements

To add self-managed nodes to your cluster, your node must meet these requirements:

• Your node must run one of the supported operating systems:
  • Ubuntu 22.04 or 24.04
  • Debian 12 or 13
  • Red Hat Enterprise Linux 9 or 10, or a rebuild such as Rocky Linux, AlmaLinux, or CentOS Stream
• You must have SSH access from a bastion host or your local workstation with root access
• Your node must have outbound internet access to the destinations in Network requirements
• Your node can be behind NAT and does not need a public IP address

Cloudfleet refuses any other release before it changes anything and shows the list of supported operating systems. The cloud-init method below is currently available for Ubuntu and Debian. For nodes in the RHEL family, use the CLI or the Terraform SSH resource.

Network requirements

Self-managed nodes only open outbound connections. They work behind NAT, need no public IP address, and need no inbound firewall rules. If your firewall restricts outbound traffic, allow these connections:

If the package mirrors of your operating system use HTTP, also allow TCP 80.

All traffic to the control plane and to other nodes goes through the encrypted WireGuard® overlay. This includes the Kubernetes API, kubectl exec, and kubectl logs, so these connections need no separate firewall rules.

Nodes connect directly to each other whenever the network allows it. If two nodes are in the same private network, they use the private network, and the traffic is still encrypted. When NAT or a firewall prevents a direct connection, the nodes connect through a relay server over TCP 443. Relayed traffic stays encrypted end to end with WireGuard®, so the relay only forwards ciphertext. A relayed connection has higher latency and lower throughput than a direct one. Inbound UDP 41641 is optional. If you allow it, more node pairs can connect directly instead of through a relay.

How a node is provisioned

With the CLI or the Terraform provider, you point Cloudfleet at a machine and Cloudfleet configures it:

1. Cloudfleet installs the Kubernetes components at the versions your cluster runs.
2. Cloudfleet prepares the host.
3. The node joins the cluster’s secure overlay network.
4. The node registers with the control plane.

The node is typically Ready about 30 seconds after the command on Ubuntu and Debian, and 40 to 80 seconds after it on the RHEL family.

You can safely repeat adding a node on the same machine. If a run stops or fails partway, run the same command (or apply the same Terraform configuration) again.

Adding self-managed nodes with Terraform

For production deployments, infrastructure-as-code workflows, or to add multiple nodes at once, use the Cloudfleet Terraform provider. The provider supports two methods:

• Cloud-init (preferred): Generates configuration that registers nodes automatically when they boot. Use this method when your target platform supports cloud-init. It gives fully automated provisioning without SSH access during setup.
• SSH: Connects to existing machines through SSH to install and configure them. Use this method for bare-metal servers, existing VMs, or platforms that do not support cloud-init.

Using Terraform with cloud-init

The Cloudfleet Terraform provider generates cloud-init userdata for any cloud provider or virtualization platform that supports cloud-init. Use this method when your infrastructure supports it. Nodes register themselves during boot, without SSH connectivity from your Terraform environment:

terraform {
  required_providers {
    cloudfleet = {
      source = "terraform.cloudfleet.ai/cloudfleet/cloudfleet"
    }
  }
}

provider "cloudfleet" {}

# Reference your existing cluster
data "cloudfleet_cfke_cluster" "cluster" {
  id = "YOUR_CLUSTER_ID"
}

# Generate cloud-init configuration for self-managed nodes
resource "cloudfleet_cfke_node_join_information" "node" {
  cluster_id = data.cloudfleet_cfke_cluster.cluster.id
  region     = "your-region"    # e.g., "eu-west-1", "on-premises"
  zone       = "your-zone"      # e.g., "datacenter-1", "rack-a"

  node_labels = {
    "cfke.io/provider" = "custom"  # Optional: identify node origin
  }

  # Optional: taints applied when the node first registers
  node_taints = [
    {
      key    = "dedicated"
      value  = "batch"
      effect = "NoSchedule"
    }
  ]
}

# The rendered cloud-init configuration is available as:
# cloudfleet_cfke_node_join_information.node.rendered
#
# Pass this to your VM's user_data field

The region and zone values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone). With these labels, you can schedule workloads to specific locations using standard node selectors. node_labels adds more labels. node_taints registers the node with taints (NoSchedule, PreferNoSchedule, or NoExecute) from the moment it joins. As a result, no workload lands on the node before you apply your tolerations. node_taints requires Terraform provider 1.5.0 or later.

Using the cloud-init output

The rendered attribute contains the cloud-init userdata. Pass it to the user_data or cloud-init field of your infrastructure provider:

# Example: Generic VM resource (syntax varies by provider)
resource "your_provider_instance" "node" {
  # ... other configuration ...

  user_data = cloudfleet_cfke_node_join_information.node.rendered
}

When the VM boots, cloud-init runs the configuration. The machine configures itself and joins your CFKE cluster with no more action from you. Terraform generates the cloud-init output for Ubuntu and Debian images.

Provider-specific guides

For complete Terraform examples including VM provisioning, firewall configuration, and security best practices, see the provider-specific guides:

For other providers that support cloud-init (for example, DigitalOcean, Linode, VMware, and OpenStack), adapt the examples above with your provider’s Terraform resources.

Adding GPU support with cloud-init

To enable NVIDIA GPU support on cloud-init provisioned nodes, add the install_nvidia_drivers option:

resource "cloudfleet_cfke_node_join_information" "gpu_node" {
  cluster_id            = data.cloudfleet_cfke_cluster.cluster.id
  region                = "your-region"
  zone                  = "your-zone"
  install_nvidia_drivers = true

  node_labels = {
    "cfke.io/provider"         = "custom"
    "cfke.io/accelerator-name" = "V100"  # Optional: specify GPU model
  }
}

Using Terraform with SSH

The cloudfleet_cfke_self_managed_node resource connects directly to existing machines through SSH and configures them as Kubernetes nodes. Use this method for bare-metal servers, existing VMs, or platforms that do not support cloud-init:

resource "cloudfleet_cfke_self_managed_node" "server" {
  cluster_id = cloudfleet_cfke_cluster.example.id
  region     = "datacenter-1"
  zone       = "rack-a"

  ssh {
    host             = "192.168.1.100"
    user             = "ubuntu"
    private_key_path = "~/.ssh/id_rsa"
    port             = 22  # Optional, defaults to 22
  }

  node_labels = {
    "environment" = "production"
  }

  # Optional: taints applied when the node first registers
  node_taints = [
    {
      key    = "dedicated"
      value  = "database"
      effect = "NoSchedule"
    }
  ]
}

The region and zone values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone). With these labels, you can schedule workloads to specific locations using standard node selectors. node_labels and node_taints work the same way as on the cloud-init resource. The SSH resource supports every operating system listed under Requirements, including the RHEL family.

For GPU nodes with SSH provisioning:

resource "cloudfleet_cfke_self_managed_node" "gpu_server" {
  cluster_id             = cloudfleet_cfke_cluster.example.id
  region                 = "datacenter-1"
  zone                   = "rack-b"
  install_nvidia_drivers = true

  ssh {
    host             = "192.168.1.101"
    user             = "ubuntu"
    private_key_path = "~/.ssh/id_rsa"
  }

  node_labels = {
    "cfke.io/accelerator-name" = "RTX-4090"
  }
}

For complete SSH resource documentation, see the Terraform resources reference.

Adding self-managed nodes with the CLI

For quick tests, single-node additions, or environments without Terraform, use the Cloudfleet CLI.

Basic CLI usage

1. Install the Cloudfleet CLI on a workstation by following the instructions here and configure it for your account.

2. Run the following command to add a self-managed node to your cluster:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --ssh-username SSH_USERNAME \
  --ssh-key SSH_KEY_LOCATION \
  --ssh-port SSH_PORT \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE

region and zone are mandatory flags that specify the location of the node. These values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone). With these labels, you can schedule workloads to specific locations using standard node selectors. You can use any string that makes sense for your infrastructure. As a best practice, use the datacenter region as the region value and the failure domain (for example, rack) as the zone value.

• You can omit the --ssh-username flag if your node uses the default username root.
• You can omit the --ssh-key flag if you have an SSH agent running and the key is added to the agent.
• You can omit the --ssh-port flag if your node uses the default SSH port 22.

The CLI connects to the machine and configures it. Then it waits for the node to register with the cluster and report Ready. The CLI shows progress on stderr. When the node is ready, the CLI prints a summary of the node:

NAME                 STATUS  READY AFTER  OS                              KERNEL          ARCH   KUBELET   RUNTIME           CPU (ALLOCATABLE)  MEMORY (ALLOCATABLE)  REGION  ZONE
cfke-test-debian-12  Ready   1m7s         Debian GNU/Linux 12 (bookworm)  6.1.0-52-amd64  amd64  v1.34.11  containerd 2.1.5  2 (1830m)          3.7 GiB (2.3 GiB)     test    debian-12

The summary respects --output-format json|yaml|table (-o). With -o json or -o yaml, the CLI prints the node as a document that scripts can consume. Progress stays on stderr. The document has the fields name, status, os, kernel, architecture, kubelet, container_runtime, cpu, cpu_allocatable, memory, memory_allocatable, region, zone, and ready_after.

• --no-wait returns as soon as the node is configured, without waiting for it to report Ready.
• --wait-timeout sets how long to wait for the node to become Ready (defaults to 3m).
• --verbose shows every command the CLI runs on the node over SSH. Use it when a step fails.

If the SSH connection fails, the CLI reports the specific cause:

• Nothing listens on the port.
• The host is unreachable or does not resolve.
• The host refuses the login for the given user and key.
• The host key does not match ~/.ssh/known_hosts. The CLI also shows the ssh-keygen -R command to clear the old entry.

3. Verify that the node is added by running:

kubectl get nodes

Adding labels and taints with the CLI

Use --label to add Kubernetes labels and --taint to register the node with taints (requires CLI 1.5.0 or later). You can repeat both flags. The CLI merges your labels with the topology labels, and a label with the same key overrides them.

Taints use the kubectl taint syntax, key=value:Effect or key:Effect. The effect is NoSchedule, PreferNoSchedule, or NoExecute. The taints apply when the node first registers, so nothing is scheduled on it before your tolerations are in place:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE \
  --label environment=production \
  --label cfke.io/provider=custom \
  --taint dedicated=database:NoSchedule

Adding GPU support with the CLI

If you have a self-managed node with an NVIDIA GPU, enable GPU support by adding the --install-nvidia-drivers flag:

cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --ssh-username SSH_USERNAME \
  --ssh-key SSH_KEY_LOCATION \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE \
  --install-nvidia-drivers

The flag installs the NVIDIA driver and configures the NVIDIA container runtime. See NVIDIA GPU configuration.

NVIDIA GPU configuration

When you enable GPU support (through Terraform or the CLI), Cloudfleet:

• Installs the NVIDIA driver that matches the cluster’s release. On Ubuntu and the RHEL family 9, setup takes about as long as for a node without a GPU. On Debian and the RHEL family 10, the driver adds a few minutes to the first boot. Debian gets the newest driver it publishes.
• Installs and configures the NVIDIA container runtime
• Labels the node with cfke.io/accelerator-manufacturer: NVIDIA
• Updates the node capacity nvidia.com/gpu field with the number of GPUs

The use of the NVIDIA drivers is subject to the NVIDIA Driver License Agreement. By using the NVIDIA drivers, you agree to the terms of the NVIDIA Driver License Agreement.

Verify GPU configuration:

kubectl get node -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity

After the node initializes, wait a few seconds for the capacity value to update.

Adding GPU model labels

When you have different GPU models in the cluster, add extra labels to identify them:

Option 1: Manual labeling

kubectl label node NODE_NAME cfke.io/accelerator-name=V100

Option 2: Automated labeling with Node Feature Discovery

The Node Feature Discovery project can label nodes automatically based on their hardware features. This Kubernetes SIG project discovers hardware features and advertises them as node labels.

For more information about GPU workloads, see GPU-based workloads and the Kubernetes GPU documentation.

Removing a self-managed node

To remove a node from a cluster:

1. Drain the node. Cloudfleet does not drain self-managed nodes for you, so run this from a machine with kubectl access to the cluster:

kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

kubectl drain cordons the node so no new Pods land on it, then evicts its Pods so they restart on other nodes. Evictions respect PodDisruptionBudgets. The --delete-emptydir-data flag deletes the data of Pods that use emptyDir volumes. For details, see Safely drain a node.

2. Run the following commands on the node. On Ubuntu and Debian:

sudo apt remove -y kubelet
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/

On the RHEL family:

sudo dnf remove -y kubelet
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/

3. Remove the node from the cluster:

kubectl delete node NODE_NAME

If you do not run this last command, the cluster garbage collector deletes the node after it becomes NotReady. However, if you remove the node manually, the Pods scheduled on it are deleted immediately and recreated on other nodes.

Moving a node to another cluster

To add a self-managed node to a different cluster, first remove it from the current cluster with the steps above. Removal unregisters the node correctly, and the node keeps no configuration from the previous cluster. Then follow the steps in this guide to add the node to the new cluster.