Node auto-provisioning

CFKE manages infrastructure provisioning for you. You don’t need to create and right-size compute nodes, delete unused ones, or manage upgrades. Cloudfleet has a built-in node auto-provisioner that calculates the optimal number and size of nodes for the current workload. You can focus on your applications instead of the infrastructure.

You can influence the node auto-provisioner’s decisions through Kubernetes Pod labels, Pod resource requests, and affinity rules. The node auto-provisioner determines the required node topology and the optimal node sizes, and provisions the nodes. When the nodes are no longer needed, the node auto-provisioner deletes them. If a node becomes unhealthy, or the cloud provider destroys it because of spot instance preemption, the node auto-provisioner replaces it with a new one.

For a comparison with traditional Kubernetes node pools, see How Cloudfleet provisions nodes. If your Pods stay in Pending and no nodes appear, see Pods stuck in Pending.

If your Pods have no specific requirements for how or where to run, CFKE can select nodes from any configured cloud provider in your Fleets. It always chooses the most cost-effective instance type. To make sure your Pods get the exact resources and type they need, use the node auto-provisioner’s layered constraints model. Reasons to constrain where your Pods run include zones with specific dependencies, specific hardware, or techniques like topology spread for high availability.

The node auto-provisioner starts when a Pod is created and no available node in the cluster can run it. The Pod then stays in Pending state. The node auto-provisioner detects this and provisions a new node that meets the Pod’s requirements. When the node is ready, the Pod is scheduled on it.

The cloud providers in your Fleets define the initial constraints for new nodes, including all available instance types, CPU architectures, zones, and purchase types. The workload specifications of your Kubernetes Pods add more constraints. Pod scheduling constraints must fall within a Fleet’s constraints. Otherwise, CFKE can’t find the right nodes and the Pods don’t deploy. For example, if a Fleet limits the number of vCPUs, you can’t schedule one more Pod that exceeds this limit.

Constraints you can request in a Pod request include:

• Resource requests: Request that a certain amount of memory or CPU is available.
• Node selection: Run on a node that has a particular label (nodeSelector).
• Node affinity: Draw a Pod to nodes with particular attributes (affinity).
• Topology spread: Use topology spread to help ensure application availability.
• Pod affinity/anti-affinity: Draw Pods towards or away from topology domains based on where other Pods are scheduled.

CFKE follows standard Kubernetes scheduling constraints, so one set of rules governs both existing and provisioned capacity.

CFKE also supports specific Well-Known Labels, Annotations, and Taints that are useful for scheduling.

Resource requests

In a Pod spec, you can set both requests and limits for the resources a Pod needs, such as CPU and memory. For example:

apiVersion: v1
kind: Pod
metadata:
  name: example-app
spec:
  containers:
  - name: example-app
    image: nginx
    resources:
      requests:
        memory: "256Mi"
        cpu: "1000m"
      limits:
        memory: "512Mi"
        cpu: "2000m"

In this example, the container requests 256MiB of memory and 1 CPU. Its limits are 512MiB of memory and 2 CPUs. The CFKE node auto-provisioner uses only requests to find the instance type for this Pod. You can also set limits to enable resource oversubscription.

See Managing Resources for Containers for the resource types that Kubernetes supports. See Specify a memory request and a memory limit for examples of memory requests.

Accelerators/GPU resources

CFKE supports the following accelerator (for example, GPU) values:

• nvidia.com/gpu

Also include a resource requirement in the workload manifest. The requirement causes the GPU-dependent Pod to be scheduled onto the appropriate node.

Here is an example of an accelerator resource in a workload manifest (for example, a Pod):

spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            nvidia.com/gpu: "1"

If an instance with an NVIDIA GPU is scheduled, CFKE installs the NVIDIA drivers on the node. For more information, see the GPU-based workloads documentation.

Selecting nodes

With nodeSelector, you can request a node that matches selected key-value pairs. These pairs can include well-known labels or custom labels that you create.

Use affinity to define more complex constraints. See Node Affinity for the complete specification.

Labels

You can use well-known labels as Pod scheduling constraints. You can also select them with nodeAffinity or nodeSelectors on your Pods.

Well-known labels

CFKE translates the following deprecated labels to their stable equivalents: failure-domain.beta.kubernetes.io/zone, failure-domain.beta.kubernetes.io/region, beta.kubernetes.io/arch, beta.kubernetes.io/os, and beta.kubernetes.io/instance-type..

The reference table at the end of this page lists all possible values for these labels.

Node selectors example

The following example uses a nodeSelector to select nodes:

nodeSelector:
  cfke.io/provider: aws
  topology.kubernetes.io/region: us-west-2

This example uses the well-known label topology.kubernetes.io/region to run the Pod on an AWS node in the us-west-2 region.

Preferences

The node auto-provisioner acts only on hard scheduling rules. When it decides which nodes to create, it ignores preferences:

• preferredDuringSchedulingIgnoredDuringExecution rules in node affinity, Pod affinity, and Pod anti-affinity.
• Topology spread constraints with whenUnsatisfiable: ScheduleAnyway.

This prevents the node auto-provisioner from creating extra nodes only to satisfy a preference. The Kubernetes scheduler still applies preferences when it places Pods on existing nodes. To make the node auto-provisioner create nodes that follow a rule, write the rule as a requirement: requiredDuringSchedulingIgnoredDuringExecution or whenUnsatisfiable: DoNotSchedule.

Node affinity

The following examples show how to use node affinity rules to include (In) and exclude (NotIn) objects. See Node affinity for more details. The following node affinity types define how hard or soft each rule is:

• requiredDuringSchedulingIgnoredDuringExecution: A hard rule. The Pod runs only on a node that matches, and the node auto-provisioner creates only matching nodes for it.
• preferredDuringSchedulingIgnoredDuringExecution: A soft rule. The Kubernetes scheduler favors existing nodes that match but places the Pod on another node if none does. The node auto-provisioner ignores this rule when it creates nodes. See Preferences.

The IgnoredDuringExecution part of each rule tells the Pod to keep running, even if conditions on the node change and the rules no longer match. You can think of these rules as required and preferred, because Kubernetes never implemented other variants.

All examples below assume that the Fleet has no constraints that prevent the use of those zones. The first constraint allows us-west-2a or us-west-2b. The second constraint allows only us-west-2b.

 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
         - matchExpressions:
           - key: "topology.kubernetes.io/zone"
             operator: "In"
             values: ["us-west-2a", "us-west-2b"]
           - key: "topology.kubernetes.io/zone"
             operator: "In"
             values: ["us-west-2b"]

If you change the second operator to NotIn, the Pod can run only in us-west-2a:

           - key: "topology.kubernetes.io/zone"
             operator: "In"
             values: ["us-west-2a", "us-west-2b"]
           - key: "topology.kubernetes.io/zone"
             operator: "NotIn"
             values: ["us-west-2b"]

To extend the example, nodeAffinity lets you define several terms. If one term doesn’t work, the next one is used. Here, if us-west-2a is not available, the second term causes the Pod to run on a spot instance in us-west-2d.

 affinity:
   nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
       nodeSelectorTerms:
         - matchExpressions: # OR
           - key: "topology.kubernetes.io/zone" # AND
             operator: "In"
             values: ["us-west-2a", "us-west-2b"]
           - key: "topology.kubernetes.io/zone" # AND
             operator: "NotIn"
             values: ["us-west-2b"]
         - matchExpressions: # OR
           - key: "karpenter.sh/capacity-type" # AND
             operator: "In"
             values: ["spot"]
           - key: "topology.kubernetes.io/zone" # AND
             operator: "In"
             values: ["us-west-2d"]

CFKE evaluates the nodeSelectorTerms in sequence and selects the first one that fulfills the requirements. If CFKE can’t provision with the first term, it tries the next ones. If all terms fail, CFKE can’t provision the Pod. CFKE then backs off and retries over a period of time. If capacity becomes available later, CFKE schedules the Pod without any action from you.

Topology spread

Use TopologySpreadConstraints in Kubernetes to spread Pods away from each other within a cluster and minimize the impact of an outage. You can think of this feature as an enhancement of Pod affinity. It associates Pods with nodes and still allows distribution.

The node auto-provisioner ignores topology spread constraints with whenUnsatisfiable: ScheduleAnyway when it creates nodes. Use DoNotSchedule, as in the example below, when you need it to create nodes that satisfy the spread. See Preferences.

For example:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/zone"
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          environment: dev
    - maxSkew: 1
      topologyKey: "kubernetes.io/hostname"
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          environment: dev

If you add this to your Pod spec, the result is:

• Pods spread across zones and hosts (topologyKey).
• The dev labelSelector includes all Pods with the label environment=dev in topology calculations. Use a selector that matches all Pods in a Deployment.
• No more than one Pod difference in the number of Pods on each host (maxSkew). For example, with three nodes and five Pods, the Pods can be spread 1, 2, 2 or 2, 1, 2 and so on. If the spread were 5 instead, Pods could be 5, 0, 0 or 3, 2, 0, or 2, 1, 2 and so on.

The node auto-provisioner supports kubernetes.io/hostname and any label from the well-known labels table as a topologyKey. Common choices are:

• topology.kubernetes.io/zone
• kubernetes.io/hostname
• topology.kubernetes.io/region
• cfke.io/provider

See Pod Topology Spread Constraints for details.

The node auto-provisioner follows zonal topology spread constraints defined for Pods that require multi-zone durability, and optimizes for compute costs. CFKE itself does not balance or rebalance availability zones for nodes.

Pod affinity/anti-affinity

To tell the CFKE node auto-provisioner your scheduling preferences, set podAffinity and podAntiAffinity in a Pod spec. These settings indicate whether Pods should be scheduled together or apart in relation to different topology domains.

The node auto-provisioner acts only on required Pod affinity and anti-affinity rules when it creates nodes. It ignores preferred rules. See Preferences.

For example:

spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: environment
            operator: In
            values:
            - production
        topologyKey: topology.kubernetes.io/zone
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: frontend
        topologyKey: kubernetes.io/hostname

The Pod affinity rule above schedules the Pod only in zones where a Pod with the label environment=production already runs.

The anti-affinity rule makes the Pod avoid any node that runs a Pod labeled app=frontend. If this anti-affinity term is on a Deployment Pod spec with a matching app=frontend label, no more than one Pod from the Deployment can run on any single node.

See Inter-Pod affinity and anti-affinity in the Kubernetes documentation for details.

Scheduling examples

The following examples show how to use the node auto-provisioner to schedule your workloads with Kubernetes Pod labels and affinity rules.

A deployment that is locked to a specific cloud provider and region

apiVersion: apps/v1
kind: Deployment
metadata:
    name: homepage
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 3
    template:
        metadata:
            labels:
                app: nginx
        spec:
            nodeSelector:
                cfke.io/provider: gcp
                topology.kubernetes.io/region: europe-west4
            containers:
                -   name: nginx
                    image: nginx:1.24
                    imagePullPolicy: Always
                    ports:
                        -   containerPort: 8080
                            name: http
                    resources:
                        requests:
                            cpu: 50m
                            memory: 56Mi
            terminationGracePeriodSeconds: 0

A deployment spread across clouds

apiVersion: apps/v1
kind: Deployment
metadata:
    name: homepage
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 3
    template:
        metadata:
            labels:
                app: nginx
        spec:
            affinity:
                podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        -   labelSelector:
                                matchExpressions:
                                    -   key: app
                                        operator: In
                                        values:
                                            - nginx
                            topologyKey: cfke.io/provider
            containers:
                -   name: nginx
                    image: nginx:1.24
                    imagePullPolicy: Always
                    ports:
                        -   containerPort: 8080
                            name: http
                    resources:
                        requests:
                            cpu: 50m
                            memory: 56Mi
            terminationGracePeriodSeconds: 0

A deployment spread across regions within a cloud

apiVersion: apps/v1
kind: Deployment
metadata:
    name: homepage
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 3
    template:
        metadata:
            labels:
                app: nginx
        spec:
            nodeSelector:
                cfke.io/provider: gcp
            affinity:
                podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        -   labelSelector:
                                matchExpressions:
                                    -   key: app
                                        operator: In
                                        values:
                                            - nginx
                            topologyKey: topology.kubernetes.io/region
            containers:
                -   name: nginx
                    image: nginx:1.24
                    imagePullPolicy: Always
                    ports:
                        -   containerPort: 8080
                            name: http
                    resources:
                        requests:
                            cpu: 50m
                            memory: 56Mi
            terminationGracePeriodSeconds: 0

A deployment spread across zones within a region

apiVersion: apps/v1
kind: Deployment
metadata:
    name: homepage
spec:
    selector:
        matchLabels:
            app: nginx
    replicas: 3
    template:
        metadata:
            labels:
                app: nginx
        spec:
            nodeSelector:
                cfke.io/provider: gcp
                topology.kubernetes.io/region: europe-west4
            affinity:
                podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        -   labelSelector:
                                matchExpressions:
                                    -   key: app
                                        operator: In
                                        values:
                                            - nginx
                            topologyKey: topology.kubernetes.io/zone
            containers:
                -   name: nginx
                    image: nginx:1.24
                    imagePullPolicy: Always
                    ports:
                        -   containerPort: 8080
                            name: http
                    resources:
                        requests:
                            cpu: 50m
                            memory: 56Mi
            terminationGracePeriodSeconds: 0

Scheduling options reference

The following table lists the well-known labels that the CFKE node auto-provisioner recognizes. You can use them in your workloads. On AWS, topology.kubernetes.io/zone holds the availability zone name of your account, for example us-west-2a. AWS maps zone names to zones differently in each account, so the table lists AWS zones by their zone ID in topology.k8s.aws/zone-id.