Proxmox

You can add Proxmox VE virtual machines as nodes in your CFKE cluster. The recommended approach uses the Cloudfleet Terraform provider with the Telmate Proxmox provider to automate VM creation with cloud-init. Terraform provisions the Proxmox VMs. A cloud-init configuration that Cloudfleet generates registers the VMs with your CFKE cluster automatically.

Prerequisites

Terraform CLI installed on your local machine.
Cloudfleet CLI configured following these setup instructions, or API token authentication configured per these guidelines.
A running Proxmox VE server with:
- API access enabled and an API token created. See the Proxmox API documentation for details.
- SSH access to the Proxmox host for uploading cloud-init configuration files.
- The snippets directory enabled on your storage (typically /var/lib/vz/snippets/).
An existing CFKE cluster. This example uses a cluster with ID CLUSTER_ID. You can also create a new cluster with Terraform by following the Terraform setup documentation.

Creating a cloud-init template

Before you deploy nodes with Terraform, create a VM template with cloud-init support. Cloud-init handles early initialization of virtual machines, including hostname configuration, SSH key injection, and package installation.

Connect to your Proxmox node with SSH to run the following commands:

This guide uses Ubuntu 24.04. The cloud-init template works with any Ubuntu or Debian release in the self-managed node requirements. Download the cloud image:

bash

wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Create a new VM that becomes your template:

bash

qm create 9000 --name ubuntu-cloudinit

Import the cloud image as the boot disk:

bash

qm set 9000 \
  --scsi0 local-lvm:0,import-from=/root/noble-server-cloudimg-amd64.img

Convert the VM to a template:

bash

qm template 9000

The template is now ready. Terraform clones it to create new nodes.

Adding Proxmox VMs to your CFKE cluster

Use the following Terraform configuration to deploy Proxmox VMs and connect them to your CFKE cluster. The configuration uploads the cloud-init file to Proxmox over SSH. It then creates a VM that joins your cluster automatically on boot.

The configuration uses the Telmate Proxmox provider. The version below was the latest at the time of writing. Check the registry for the most recent version. See the provider documentation for more configuration options.

Terraform providers

terraform

terraform {
  required_providers {
    proxmox = {
      source  = "telmate/proxmox"
      version = "3.0.2-rc06"
    }
    cloudfleet = {
      source = "terraform.cloudfleet.ai/cloudfleet/cloudfleet"
    }
  }
}

# Cloudfleet provider authenticates using one of these methods (in priority order):
# 1. Environment variables: CLOUDFLEET_ORGANIZATION_ID, CLOUDFLEET_ACCESS_TOKEN_ID, CLOUDFLEET_ACCESS_TOKEN_SECRET
# 2. Direct configuration: organization_id, access_token_id, access_token_secret
# 3. CLI profile: Uses the 'default' profile created with 'cloudfleet auth add-profile'
provider "cloudfleet" {}

provider "proxmox" {
  pm_api_url          = var.proxmox_api_url
  pm_api_token_id     = var.proxmox_api_token_id
  pm_api_token_secret = var.proxmox_api_token_secret
}

The Cloudfleet provider supports multiple authentication methods. The simplest approach is to create a profile with the Cloudfleet CLI (cloudfleet auth add-profile). The CLI stores profiles in credentials.json in the cloudfleet folder of your user configuration directory, for example ~/.config/cloudfleet/credentials.json on Linux. For CI/CD pipelines, use environment variables or API tokens. See the Terraform provider documentation for more details.

Variables

The configuration needs variables to connect to Cloudfleet and Proxmox. Create a terraform.tfvars file, or pass the values on the command line when you run terraform apply.

terraform

# Cloudfleet configuration
variable "cfke_cluster_id" {
  type        = string
  description = "The unique identifier of your CFKE cluster. Find this in the Cloudfleet console or via 'cloudfleet clusters list'."
}

variable "node_region" {
  type        = string
  default     = "on-premises"
  description = "Geographic region label for the node. Maps to the topology.kubernetes.io/region Kubernetes label. Use a meaningful identifier like your datacenter location (e.g., 'frankfurt', 'us-east')."
}

variable "node_zone" {
  type        = string
  default     = "proxmox"
  description = "Availability zone label for the node. Maps to the topology.kubernetes.io/zone Kubernetes label. Use a meaningful identifier like your rack or failure domain (e.g., 'rack-1', 'building-a')."
}

# Proxmox connection
variable "proxmox_api_url" {
  type        = string
  description = "Full URL to the Proxmox API endpoint, including port and path (e.g., 'https://proxmox.example.com:8006/api2/json')."
}

variable "proxmox_api_token_id" {
  type        = string
  sensitive   = true
  description = "Proxmox API token ID in the format 'user@realm!tokenname' (e.g., 'root@pam!terraform'). Create this in Proxmox under Datacenter > Permissions > API Tokens."
}

variable "proxmox_api_token_secret" {
  type        = string
  sensitive   = true
  description = "The secret value of your Proxmox API token. This is shown only once when creating the token."
}

variable "proxmox_ssh_host" {
  type        = string
  description = "Hostname or IP address of your Proxmox server for SSH connections. Used to upload cloud-init configuration files to the snippets directory."
}

variable "proxmox_ssh_private_key" {
  type        = string
  default     = "~/.ssh/id_ed25519"
  description = "Path to the SSH private key for connecting to the Proxmox host."
}

variable "proxmox_target_node" {
  type        = string
  description = "Name of the Proxmox node where the VM will be created. This is the node name shown in the Proxmox UI, not the hostname."
}

variable "proxmox_storage" {
  type        = string
  default     = "local-lvm"
  description = "Proxmox storage pool for VM disks and cloud-init drive. Must have sufficient space for the VM disk size."
}

# VM configuration
variable "vm_clone_template" {
  type        = string
  default     = "ubuntu-cloudinit"
  description = "Name of the VM template to clone. Must match the template created in the 'Creating a cloud-init template' section."
}

variable "node_count" {
  type        = number
  default     = 1
  description = "Number of CFKE nodes to create."
}

Cloudfleet node join configuration

Next, generate the cloud-init configuration that tells the VM how to join your CFKE cluster. The cloudfleet_cfke_node_join_information resource fetches the certificates, tokens, and bootstrap scripts from Cloudfleet. It renders them as a cloud-init YAML file.

The region and zone values become Kubernetes node labels (topology.kubernetes.io/region and topology.kubernetes.io/zone). You can use these labels for workload placement and topology-aware scheduling.

terraform

data "cloudfleet_cfke_cluster" "cluster" {
  id = var.cfke_cluster_id
}

resource "cloudfleet_cfke_node_join_information" "proxmox" {
  cluster_id    = data.cloudfleet_cfke_cluster.cluster.id
  region        = var.node_region
  zone          = var.node_zone
  gzip          = false
  base64_encode = false

  node_labels = {
    "cfke.io/provider" = "proxmox"
  }
}

Cloud-init file upload

Many cloud providers accept cloud-init data directly through an API. Proxmox instead needs cloud-init configuration files in the snippets directory on the host filesystem. The following resource uses SSH to upload the generated configuration to your Proxmox server.

The triggers block uploads the file again whenever the cloud-init content changes, for example when certificates rotate or the cluster configuration changes.

terraform

resource "null_resource" "cloud_init_user_data" {
  triggers = {
    content = cloudfleet_cfke_node_join_information.proxmox.rendered
  }

  connection {
    type        = "ssh"
    host        = var.proxmox_ssh_host
    user        = "root"
    private_key = file(var.proxmox_ssh_private_key)
  }

  provisioner "file" {
    content     = cloudfleet_cfke_node_join_information.proxmox.rendered
    destination = "/var/lib/vz/snippets/${var.cfke_cluster_id}.yaml"
  }
}

Proxmox VMs

Finally, create the VMs by cloning the template that you prepared earlier. Each node gets a unique random suffix, so naming stays consistent when you scale up or down. The random_id resource generates stable identifiers that persist across Terraform runs.

The cicustom parameter references the cloud-init file from the previous step. Proxmox injects this file into the VM during boot. When the VM starts, cloud-init runs the bootstrap script, installs the Kubernetes components, and registers the node with your CFKE cluster. This process typically takes 2-3 minutes.

Adjust the CPU, memory, and disk size to match your workload requirements.

terraform

resource "random_id" "node" {
  for_each    = toset([for i in range(1, var.node_count + 1) : tostring(i)])
  byte_length = 4
  keepers = {
    cluster_id = cloudfleet_cfke_node_join_information.proxmox.cluster_id
  }
}

resource "proxmox_vm_qemu" "cfke_node" {
  for_each    = random_id.node
  depends_on  = [null_resource.cloud_init_user_data]
  name        = "cfke-node-${each.value.hex}"
  target_node = var.proxmox_target_node
  agent       = 1

  # Compute resources
  cpu {
    cores   = 4
    sockets = 1
  }
  memory = 8096

  # Clone from template
  clone  = var.vm_clone_template
  boot   = "order=scsi0"
  scsihw = "virtio-scsi-single"

  # Cloud-init configuration
  cicustom  = "vendor=local:snippets/${var.cfke_cluster_id}.yaml"
  ipconfig0 = "ip=dhcp,ip6=auto"

  serial {
    id = 0
  }

  # Storage
  disks {
    scsi {
      scsi0 {
        disk {
          storage = var.proxmox_storage
          size    = "20G"
        }
      }
    }
    ide {
      ide1 {
        cloudinit {
          storage = var.proxmox_storage
        }
      }
    }
  }

  # Network
  network {
    id     = 0
    bridge = "vmbr0"
    model  = "virtio"
  }
}

Run these commands to provision your Proxmox infrastructure and add the VMs to your CFKE cluster:

bash

terraform init
terraform apply

After provisioning completes, check that your new nodes joined the cluster:

bash

kubectl get nodes

Proxmox-specific considerations

Cloud-init snippet storage

Proxmox needs cloud-init files in the snippets directory. Make sure that the snippets content type is enabled on your storage:

In the Proxmox UI, go to Datacenter > Storage
Select your storage pool
Under Content, make sure that Snippets is enabled

You can also enable it on the command line:

bash

pvesm set local --content images,rootdir,vztmpl,backup,iso,snippets

SSH key configuration

The Terraform configuration uses SSH to upload cloud-init files to Proxmox. Make sure that your SSH key is authorized on the Proxmox host. The default key path is ~/.ssh/id_ed25519. To use a different key, set the proxmox_ssh_private_key variable.

Scaling nodes

To add more nodes, update the node_count variable in your terraform.tfvars:

hcl

node_count = 3

Then apply the changes:

bash

terraform apply

Each new node receives a unique random suffix (for example, cfke-node-a1b2c3d4). The random_id resource keeps these suffixes stable across Terraform runs. Scaling up or down does not affect existing nodes.

Firewall configuration

Nodes only open outbound connections and do not need inbound access from the internet. If your Proxmox firewall restricts outbound traffic, allow the destinations listed in Network requirements. The most important ones are TCP 443 to the coordination server and the relay servers, UDP 3478 for NAT discovery, and UDP from source port 41641 for direct WireGuard® connections between nodes.

Next steps

Learn about self-managed nodes and their requirements
Configure on-premises load balancing with BGP for exposing services
Explore the Terraform provider documentation for advanced configurations

Previous
← Exoscale