Pods stuck in Pending

When you deploy a workload and no suitable node exists, Cloudfleet’s node auto-provisioner creates one. This normally takes 2-4 minutes. If your pods stay in Pending longer than that, work through this checklist from top to bottom. Each step matches a failure mode we see regularly in support tickets.

Step 1: Read the scheduling events

Start with the events on the pod:

bash 
kubectl describe pod POD_NAME

Look at the FailedScheduling message. It usually names the blocking condition, such as Insufficient memory, an unsatisfiable node selector, or no nodes available to schedule pods.

Then check whether the auto-provisioner has created a node request. Every node request is represented by a NodeClaim object:

bash 
kubectl get nodeclaims -o wide

Three outcomes are possible:

  • No NodeClaim appears: the auto-provisioner cannot find any instance type that satisfies your pod’s constraints and the Fleet’s configuration. Continue with step 2.
  • A NodeClaim exists but stays Registered=Unknown with reason NodeNotFound: the provider was asked to create a machine, but no machine ever joined the cluster. Continue with step 3.
  • A NodeClaim becomes ready but the pod still does not fit: the pod’s resource requests are the problem. Continue with step 4.

Step 2: No instance type matches your constraints

If the cluster events or the console Events tab show messages like no instance type met the scheduling requirements, the combination of your pod’s scheduling constraints and the Fleet configuration excludes every available machine. Common causes:

  • An exact instance type pin that the location does not offer. Cloud providers do not offer every instance type in every zone or region. A nodeSelector such as node.kubernetes.io/instance-type: cpx31 combined with a specific zone fails permanently if that model is not sold there. Pin the instance family instead and let Cloudfleet pick a concrete size:

    yaml 
    nodeSelector:
  cfke.io/instance-family: cpx

    In general, avoid pinning exact instance types. Set realistic resource requests and let the auto-provisioner select the most cost-effective match.

  • Invalid region values in the Fleet configuration. Region names are provider-specific. For example, AWS-style names such as eu-central-1 match nothing on a Hetzner Fleet, whose locations are fsn1, nbg1, or hel1. A Fleet constrained to nonexistent regions provisions nothing. Review your Fleet constraints and the valid values in Node regions.

  • Over-constrained combinations. An instance type pin plus a zone pin plus pod anti-affinity can be individually satisfiable but jointly impossible. Relax one dimension at a time to find the conflict.

  • The Fleet’s vCPU limit is reached. Fleets can cap the total vCPU count as a cost control. When the cap is reached, new pods stay Pending until you raise the limit, add another Fleet, or remove workloads.

Step 3: The provider cannot create machines

If NodeClaims are created but no machine ever registers, the provider is rejecting or failing the launch. Check the Events tab in the Cloudfleet Console for provider errors, then verify:

  • Fleet credentials are valid and have write permission. This is the most common cause. A Hetzner API token that was revoked, regenerated, or created with read-only permission causes every node creation to fail with Unauthorized (401) or permission denied (403), while existing nodes keep serving traffic and mask the problem, sometimes for weeks. Create a fresh token with “Read & Write” permission and update it on the Fleet in the Cloudfleet Console. The fix takes effect within minutes and does not affect running workloads.

  • The provider account is in good standing. Providers restrict accounts to read-only or suspend network access over unpaid invoices, pending identity verification, or abuse reviews. In that state Cloudfleet’s API calls fail even with a valid token. Check your provider console for notices. Once the restriction is lifted, the auto-provisioner recovers on its own.

  • Required provider APIs are enabled. On GCP, the Compute Engine API must be enabled in the project before any machine can be created:

    bash 
    gcloud services enable compute.googleapis.com --project PROJECT_ID

  • Provider quotas are sufficient. Newly created cloud accounts often have low default quotas for vCPUs or instances per region. On Hetzner, hitting the account’s server limit surfaces as server limit reached in the provisioning events; raise the limit in the Hetzner console. Other providers return similar quota errors. Request an increase in the provider console, and provisioning resumes automatically.

Step 4: Check your resource requests

On CFKE, resource requests are not just a scheduling detail: they drive node sizing, workload placement, and node stability. Workloads without realistic requests and limits are the most common root cause of unstable clusters, from scheduling loops to frozen nodes. The auto-provisioner sizes nodes from your pods’ resource requests, so implausible requests produce confusing failures:

  • Missing or very small requests make Cloudfleet pick the smallest available instance. System components can consume nearly all of a small node, leaving no room for your pod, which triggers a loop of nodes being added and removed. Set realistic CPU and memory requests on all workloads, or constrain the Fleet to a minimum instance size.

  • Impossibly large requests have the opposite effect. A misconfigured VerticalPodAutoscaler, for example, can set a container’s memory request to an effectively unbounded value, making the scheduler report Insufficient memory on a nearly empty node. Compare the pod’s requests against the node’s allocatable resources:

    bash 
    kubectl get pod POD_NAME -o jsonpath='{.spec.containers[*].resources.requests}'
kubectl describe node NODE_NAME | grep -A 6 Allocatable

    Fix the request (and any VPA maxAllowed policy), then recreate the pod. Restarting the pod without fixing the source reapplies the bad value.

  • Single-architecture container images crash-loop when Cloudfleet provisions a cheaper ARM node. See Handle single-platform container images.

Things that make it worse

  • Never rename or modify Cloudfleet-provisioned machines in the provider console. Cloudfleet tracks machines by their creation name; a renamed server is treated as lost, producing phantom node churn and orphaned machines. Make all capacity changes through Cloudfleet.
  • Never delete NodeClaims manually. Deleting a NodeClaim removes its node; the system pods running there become unschedulable and trigger replacement provisioning, which can look like runaway node creation.

Still stuck?

If the checklist does not surface the cause, check the Events tab in the Cloudfleet Console for provisioning errors and contact support with your cluster ID and the pod’s FailedScheduling events.

Next
Cannot reach the cluster API →
On this page