Fleet taints

Fleet taints keep workloads off a Fleet. Every node the Fleet provisions carries the taints, so only Pods that tolerate them can run there. Use them to dedicate a Fleet to one kind of workload, for example CI runners or GPU jobs, next to the Fleet that runs your production services.

Why use Fleet taints

A nodeSelector pins a workload to a Fleet, but it does not keep other workloads off it. Without taints, any Pod without scheduling rules can land on your dedicated nodes. Tainting nodes by hand with kubectl taint does not help either. Nodes that the node auto-provisioner creates later do not carry the taint. The node auto-provisioner also does not know about it when it plans capacity.

Fleet taints are part of the Fleet, so every node gets them from the moment it joins the cluster. The node auto-provisioner only provisions nodes in that Fleet for Pods that tolerate them.

How taints work

A taint has a key, an optional value, and an effect:

  • NoSchedule: Pods without a matching toleration are not scheduled on the node.
  • PreferNoSchedule: the scheduler avoids the node for Pods without a matching toleration, but uses it if nothing else fits.
  • NoExecute: like NoSchedule, and Pods already running on the node without a matching toleration are evicted.

The CFKE system components that run on every node, such as the CNI, DNS, and the connectivity agent, tolerate all taints, so they keep working on tainted nodes. For background, see Taints and Tolerations in the Kubernetes documentation.

Keys under node.kubernetes.io, node.cloudprovider.kubernetes.io, node.cilium.io, karpenter.sh, and cfke.io (and their subdomains) are reserved, because Kubernetes and CFKE set those taints themselves. A Fleet can have up to 50 taints, and each key and effect pair can appear once.

Running workloads on a tainted Fleet

To run a workload on a tainted Fleet, add a toleration for the taint. To also keep the workload on that Fleet only, add a node selector on the karpenter.sh/nodepool label, which carries the Fleet name. For a Fleet named runners with the taint dedicated=ci:NoSchedule:

yaml 
spec:
  nodeSelector:
    karpenter.sh/nodepool: runners
  tolerations:
    - key: dedicated
      operator: Equal
      value: ci
      effect: NoSchedule

Configuring Fleet taints

Set taints when you create or update a Fleet:

  • Console: turn on Taint nodes in the Fleet wizard and add a row per taint.

  • API and CLI: set the taints list on the Fleet:

    json 
    {
  "taints": [
    { "key": "dedicated", "value": "ci", "effect": "NoSchedule" }
  ]
}

  • Terraform: set the taints attribute on the cloudfleet_cfke_fleet resource.

Fleet updates replace the whole Fleet configuration, so an update without taints removes all taints from the Fleet. When you change a Fleet’s taints, existing nodes no longer match the Fleet, and CFKE replaces them with nodes that carry the new taints, within the Fleet’s disruption limits.

Previous
← Fleet constraints
Next
Auto-provisioning profiles →
On this page