Control plane pressure and API throttling

The Kubernetes API server protects itself under load with API Priority and Fairness: requests are queued per flow and rejected when queues overflow. On CFKE, the most common source of control plane pressure is not application traffic but platform tooling: GitOps controllers reconciling too often, operators re-listing large object sets, and tools that store large objects in the cluster. This page shows how to recognize the condition and the fixes that resolved real support cases.

This page complements Kubernetes API server quotas, which covers hard object size and data store limits.

Recognizing control plane pressure

Typical symptoms, often appearing together:

  • controllers receive HTTP 429 (throttled) or intermittent 500/504 responses
  • multiple controllers on different nodes lose leader election at the same time and restart: Failed to renew lease ... context deadline exceeded
  • kubectl feels slow while /readyz reports healthy
  • watch connections close repeatedly and controllers log reconnects

Confirm with the API server’s flow control metrics:

bash 
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
kubectl get --raw /metrics | grep apiserver_flowcontrol_current_inqueue_requests

Non-zero rejections with reason="time-out", typically in the service-accounts or workload-low priority levels, mean requests are timing out in the queue: the control plane is saturated by request volume, not broken.

Fix reconcile storms from GitOps tooling

A single misconfiguration can generate thousands of API requests per hour. Work through the tools you run:

Flux:

  • Raise spec.interval on HelmReleases and Kustomizations for stable releases to 10 minutes or more. One-minute intervals re-list everything they manage every 60 seconds.

  • Fix permanently failing reconciliations; a Kustomization that fails every minute re-applies the entire repository every minute. A common example is a SOPS-encrypted secret without a decryption block on the root Kustomization (Secret/... is SOPS encrypted, configuring decryption is required):

    yaml 
    spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

  • After committing fixes, resume anything that was suspended: flux resume kustomization NAME -n flux-system.

Argo CD:

  • Remove empty parameters: [] blocks under spec.source.helm; Argo CD treats them as a permanent diff and performs a full hard refresh (git re-clone and complete state read) on every sync cycle.
  • Make exactly one Application own cluster-scoped CRDs. Multiple Applications installing the same chart at different versions rewrite the CRDs on every sync, generating tens of thousands of updates and closing everyone else’s watch connections.

Operators:

  • Set sensible reconcile periods (for example the helm.sdk.operatorframework.io/reconcile-period annotation, or the operator’s own interval setting) instead of the default of about one minute.
  • Avoid seconds-level spec.interval on custom resources; 28 resources re-checked every 3 seconds is a request storm.
  • Scope kube-state-metrics with --resources so it does not watch secrets, configmaps, and leases it does not need.
  • If controllers lose leases under load, widen the leader election settings (leaseDuration, renewDeadline, retryPeriod) and stagger resync intervals so operators do not re-list in lockstep.

Fix object bloat

The API server keeps stored objects in memory, so tools that write many large objects degrade the whole control plane:

  • Trivy Operator: SBOM generation creates one large report object per workload revision. Disable it if you do not consume SBOMs (OPERATOR_SBOM_GENERATION_ENABLED=false), delete existing sbomreports and clustersbomreports, and set scannerReportTTL. Vulnerability scanning is unaffected.
  • Velero: frequent full-cluster filesystem backups with long retention accumulate thousands of backup objects. Reduce schedule frequency and retention.
  • Log collectors: do not pull logs through the Kubernetes API from every node. Configure agents such as Grafana Alloy to discover only pods on their own node and to tail log files from the host (/var/log/pods) instead of the API.

“Forbidden: this resource is crucial” errors

Symptom: a reconciler loops with errors like:

daemonsets.apps "cilium" is forbidden: ... This resource is crucial for correct
functioning of your cluster. You are not allowed to create, update, or delete
this resource in CFKE.

Cause: CFKE write-protects the platform components it manages (the CNI, metrics-server, cloud controller, node provisioning objects, and admission webhooks). Tools like Rancher, Flux, or Helm that try to adopt or reinstall them are rejected by design.

Fix: remove the duplicate installation from your GitOps configuration; the platform-managed copy already provides the function (for example, kubectl top and HPA work out of the box via the managed metrics-server). Do not attempt to grant the tool write access. The errors are harmless to the cluster but generate steady API load and log noise; if you cannot remove the source quickly, filter the messages in your log pipeline. The same applies to node provisioning objects: do not manage NodePool or NodeClass objects with your own charts; define Fleets instead.

When capacity is the answer

Basic clusters share control plane resources with other customers, and heavy controller stacks (GitOps, policy engines, monitoring, backup, service catalogs) can outgrow them even after tuning. Pro clusters run dedicated, multi-replica API servers that sustain significantly higher request volume. If flow control rejections persist after the fixes above, consider upgrading the cluster tier or splitting operator-heavy stacks across clusters.

Previous
← Load balancer issues
Next
Self-managed node issues →
On this page