Fleet configuration

Fleets represent cloud accounts connected to a CFKE cluster. When you create a Fleet, CFKE automatically provisions nodes in that cloud account to run your workloads. CFKE currently supports AWS, GCP, and Hetzner for node auto-provisioning.

To create a Fleet, grant CFKE the permissions it needs to provision and manage nodes in your cloud account.

Cloudfleet orchestrates critical infrastructure for its customers, so security is a top priority. We always follow the principle of least privilege and request only the permissions needed to operate the service. Depending on your setup, these permissions may include launching and terminating instances and managing load balancers.

For infrastructure providers with mature identity and access management, we use passwordless and credential-less authentication. For other providers, we encrypt and protect credentials and store them securely in your own cluster. In both cases, only your cluster has access to your cloud provider accounts.

For most cloud providers (for example, AWS and GCP), you do not need to share hardcoded credentials with us. Instead, you delegate permissions in your cloud account to a role or service account that we manage. You do not need to store and rotate static credentials. You can also restrict the permissions of Cloudfleet to a specific set of actions and resources. Cloudfleet then cannot access any other resources in your account and can only perform the actions needed to deliver infrastructure for CFKE.

You can revoke the permissions of Cloudfleet at any time. We then immediately lose access to your account.

Quick start with Terraform

The Cloudfleet Terraform provider is the fastest way to set up your cloud account and create a Fleet. The following Terraform configuration creates a CFKE cluster and a Fleet for AWS, GCP, and Hetzner Cloud. On GCP, the example sets up the required permissions with a custom role. On AWS, it uses the CFKE Fleet Terraform module for AWS. The module sets up the IAM roles, the trust policies, and the VPCs in each region.

terraform 
terraform {
    required_providers {
        cloudfleet = {
            source = "terraform.cloudfleet.ai/cloudfleet/cloudfleet"
        }
    }
}

variable "cfke_control_plane_region" {
    description = "CFKE control plane region where the cluster is deployed"
    type        = string
    default     = "europe-central-1a"
}

variable "gcp_project" {
    type        = string
    description = "GCP project ID where CFKE nodes will be provisioned"
}

variable "hetzner_api_key" {
    description = "API key for Hetzner Cloud"
    type        = string
    sensitive   = true
}

variable "aws_region" {
    description = "AWS region where CFKE nodes will be provisioned"
    type        = string
    default     = "eu-central-1"
}

variable "aws_profile" {
    description = "AWS profile to use to authenticate with AWS"
    type        = string
    default     = "default"
}

provider "aws" {
    region  = var.aws_region
    profile = var.aws_profile
}

provider "cloudfleet" {
    profile = "default"
}

resource "cloudfleet_cfke_cluster" "cfke_test" {
    name   = "cfke-test"
    region = var.cfke_control_plane_region
    tier   = "basic"
}

resource "google_project_iam_custom_role" "cfke_node_autoprovisioner" {
    project = var.gcp_project
    permissions = [
        // Node management permissions
        "compute.instances.create",
        "compute.instances.delete",
        "compute.instances.get",
        "compute.instances.list",
        "compute.disks.create",
        "compute.subnetworks.use",
        "compute.subnetworks.useExternalIp",
        "compute.instances.setMetadata",
        "compute.instances.setTags",
        "compute.instances.setLabels",

        // Load Balancer management permissions (required for Load Balancing, can be omitted if not used)
        "compute.firewalls.create",
        "compute.firewalls.delete",
        "compute.firewalls.get",
        "compute.firewalls.update",
        "compute.regionHealthChecks.get",
        "compute.regionHealthChecks.create",
        "compute.regionHealthChecks.update",
        "compute.regionHealthChecks.delete",
        "compute.regionHealthChecks.useReadOnly",
        "compute.networkEndpointGroups.get",
        "compute.networkEndpointGroups.delete",
        "compute.networkEndpointGroups.create",
        "compute.networkEndpointGroups.use",
        "compute.networkEndpointGroups.attachNetworkEndpoints",
        "compute.networkEndpointGroups.detachNetworkEndpoints",
        "compute.instances.use",
        "compute.zoneOperations.get",
        "compute.regionBackendServices.get",
        "compute.regionBackendServices.create",
        "compute.regionBackendServices.update",
        "compute.regionBackendServices.delete",
        "compute.forwardingRules.get",
        "compute.forwardingRules.create",
        "compute.forwardingRules.update",
        "compute.forwardingRules.delete",
        "compute.forwardingRules.setLabels",
        "compute.networks.updatePolicy",
        "compute.zones.list"
    ]
    role_id = "cfke.nodeAutoprovisioner"
    title   = "CFKE Node-autoprovisioner"
}

resource "google_project_iam_binding" "gcp_project_binding" {
    project = var.gcp_project
    role    = google_project_iam_custom_role.cfke_node_autoprovisioner.id
    members = [
        "principal://iam.googleapis.com/projects/207152264238/locations/global/workloadIdentityPools/cfke/subject/${cloudfleet_cfke_cluster.cfke_test.id}"
    ]
}

module "cfke_connected_fleet" {
    source               = "registry.terraform.io/cloudfleetai/cfke-connected-fleet/aws"
    version              = "~> 1"
    control_plane_region = cloudfleet_cfke_cluster.cfke_test.region
    cluster_id           = cloudfleet_cfke_cluster.cfke_test.id
}

resource "cloudfleet_cfke_fleet" "fleet" {

    depends_on = [
        google_project_iam_binding.gcp_project_binding
    ]

    cluster_id = cloudfleet_cfke_cluster.cfke_test.id
    name       = "cfke-multi-cloud-fleet"

    limits {
        cpu = 24
    }

    hetzner {
        api_key = var.hetzner_api_key
    }

    aws {
        role_arn = module.cfke_connected_fleet.fleet_arn
    }

    gcp {
        project_id = var.gcp_project
    }
}

Cloud provider specific setup

On every cloud provider, CFKE provisions nodes in a network that it creates or that already exists by default. You can also run a Fleet in a custom network that you manage. Examples are your own VPC and subnets on AWS, a custom or Shared VPC network on GCP, or an existing network on Hetzner Cloud. Use a custom network when nodes must share a private network with your other resources, use your own address ranges, or follow your network and firewall topology. Cloudfleet support sets up custom networks. To use one, contact Cloudfleet support.

AWS

To connect an AWS account with CFKE, you do not need hardcoded credentials. Cloudfleet internally manages a unique AWS IAM role for each CFKE cluster. To authorize CFKE in your AWS account, create a role and attach the required permissions. Then configure a trust policy that allows CFKE’s internal IAM role to assume your role.

This method requires more IAM resources and may seem complex, but it is more secure than hardcoded credentials. Cloudfleet provides a Terraform module that creates the required IAM role and policies automatically. You can find the CFKE Fleet Terraform module here.

The module also deploys VPCs and subnets in every region supported by CFKE. These resources are free of charge. To use your own VPC and subnets instead, contact Cloudfleet support.

The module creates IAM permissions that restrict access to specific virtual machines. The IAM policies apply only to resources with certain tags. Cloudfleet can create and delete EC2 instances only if they have a specific tag. As a result, Cloudfleet operates in your AWS account but cannot use its IAM permissions to create or delete any resources without those tags.

By default, CFKE nodes on AWS get public IP addresses to communicate with the control plane, download packages, and interact with other nodes. Security groups block inbound traffic from the internet. CFKE nodes only need outbound connections, so they can also run in private subnets without public IP addresses, behind a NAT gateway. To set up private nodes, contact Cloudfleet support. For the outbound destinations that nodes need, see Network requirements.

  1. Use the module to create the required IAM roles and policies in your AWS account:

    terraform 
    module "cfke_connected_fleet" {
  source               = "registry.terraform.io/cloudfleetai/cfke-connected-fleet/aws"
  version              = "~> 1"
  control_plane_region = "CONTROL_PLANE_REGION"
  cluster_id           = "CLUSTER_ID"
}

    Replace CONTROL_PLANE_REGION with your CFKE control plane region and CLUSTER_ID with your cluster ID. You can find both values in the Cloudfleet Console.

  2. The module outputs fleet_arn. Use this ARN when you create a Fleet for AWS in the Cloudfleet Console.

For more details, see the module documentation.

GCP

CFKE uses Workload Identity Federation to access your GCP project without hardcoded credentials. Each CFKE cluster has a unique principal managed internally by Cloudfleet. To authorize CFKE in your GCP project, grant the roles/compute.instanceAdmin.v1 role to this principal.

To provision CFKE nodes in your GCP project:

  1. By default, CFKE uses the default VPC network of your GCP project. Make sure this network exists with subnet creation mode set to Automatic. With this setting, subnets exist in all regions.

    To use a custom VPC network or a Shared VPC network instead, contact Cloudfleet support.

    To check VPC networks with the gcloud CLI:

    bash 
    gcloud compute networks list --project=YOUR_PROJECT_ID

    The output looks like this:

    bash 
    NAME     SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4  INTERNAL_IPV6_RANGE
default  AUTO         REGIONAL

    If you do not get this result, create a new auto-subnetted VPC:

    bash 
    gcloud compute networks create default \
--subnet-mode=auto \
--project=YOUR_PROJECT_ID

  2. By default, CFKE nodes on GCP get public IP addresses to communicate with the control plane, download packages, and interact with other nodes. VPC firewall rules block inbound traffic from the internet. Some organizations restrict public IP addresses with the constraints/compute.vmExternalIpAccess policy. With the default setup, make sure this policy is not set to DENY.

    CFKE nodes only need outbound connections, so they can also run without public IP addresses, behind Cloud NAT. To set up private nodes, contact Cloudfleet support. For the outbound destinations that nodes need, see Network requirements.

    To check organization policies with the gcloud CLI:

    bash 
    gcloud org-policies list --project=YOUR_PROJECT_ID

    If you have the necessary permissions, you can remove the DENY restriction with the following command.

    WARNING: This command re-enables the use of public IP addresses. Proceed according to your security policy.

    bash 
    gcloud org-policies reset constraints/compute.vmExternalIpAccess \
--project=YOUR_PROJECT_ID

  3. Grant the roles/compute.instanceAdmin.v1 role to the following principal:

    principal://iam.googleapis.com/projects/207152264238/locations/global/workloadIdentityPools/cfke/subject/CLUSTER_ID

    Use the gcloud CLI to grant the role:

    bash 
    gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=principal://iam.googleapis.com/projects/207152264238/locations/global/workloadIdentityPools/cfke/subject/CLUSTER_ID \
  --role=roles/compute.instanceAdmin.v1

    Replace PROJECT_ID with your GCP project ID and CLUSTER_ID with your cluster ID.

    To grant more granular permissions instead of the built-in role, create a custom IAM role with the following permissions:

    Node management permissions:

    • compute.instances.create
    • compute.instances.delete
    • compute.instances.get
    • compute.instances.list
    • compute.disks.create
    • compute.subnetworks.use
    • compute.subnetworks.useExternalIp
    • compute.instances.setMetadata
    • compute.instances.setTags
    • compute.instances.setLabels

    Load Balancer management permissions (required for Load Balancing, can be omitted if not used):

    • compute.firewalls.create
    • compute.firewalls.delete
    • compute.firewalls.get
    • compute.firewalls.update
    • compute.regionHealthChecks.get
    • compute.regionHealthChecks.create
    • compute.regionHealthChecks.update
    • compute.regionHealthChecks.delete
    • compute.regionHealthChecks.useReadOnly
    • compute.networkEndpointGroups.get
    • compute.networkEndpointGroups.delete
    • compute.networkEndpointGroups.create
    • compute.networkEndpointGroups.attachNetworkEndpoints
    • compute.networkEndpointGroups.detachNetworkEndpoints
    • compute.networkEndpointGroups.use
    • compute.instances.use
    • compute.zoneOperations.get
    • compute.regionBackendServices.get
    • compute.regionBackendServices.create
    • compute.regionBackendServices.update
    • compute.regionBackendServices.delete
    • compute.forwardingRules.get
    • compute.forwardingRules.create
    • compute.forwardingRules.update
    • compute.forwardingRules.delete
    • compute.forwardingRules.setLabels
    • compute.networks.updatePolicy
    • compute.zones.list

After you create the IAM binding, enter the GCP Project ID in the Fleet creation wizard in the Cloudfleet Console.

Cloudfleet does not access any resources that CFKE did not provision. If your setup requires additional isolation, create a separate Google Cloud project for CFKE. The role assigned to Cloudfleet then has access only to the resources in that project.

Hetzner Cloud

  1. Follow the Hetzner Cloud documentation to generate an API token with “Read & Write” permissions.

  2. Enter this token in the Cloudfleet Console when you create a Fleet for Hetzner Cloud.

By default, CFKE creates a separate network per region in your Hetzner Cloud account for provisioning nodes. This network is named cfke-CLUSTER_ID-NETWORK_REGION_NAME.

You can also attach a Fleet to an existing Hetzner Cloud network that you manage, instead of letting CFKE create one. Use this option when nodes must share a private network with other workloads, share routes with customer-managed virtual machines, or follow your own network and firewall topology. To enable a custom network for your Hetzner Cloud Fleet, contact Cloudfleet support.

Hetzner Cloud does not currently support granular permissions for API tokens, so you cannot restrict a token to specific resources. Cloudfleet does not access any resources that CFKE did not provision. If your setup requires additional isolation, create a separate Hetzner Cloud project for CFKE. The API token that Cloudfleet uses then has access only to the resources in that project.

Previous
← Node regions
On this page