Kubernetes API server quotas

To keep your cluster healthy and predictable, the Cloudfleet Kubernetes API server enforces two quotas: a per-object size limit and an overall data store size limit. Both quotas protect the control plane from runaway growth, which can make a cluster slow, unstable, or impossible to recover.

Per-object size limit (1 MiB)

Any single Kubernetes object you create or update, such as a ConfigMap, Secret, CustomResource, or large annotation set, must be 1 MiB or smaller. The API server rejects requests that exceed this limit.

If you see an error similar to:

Request entity too large: limit is ...

or your client reports ResourceExhausted, the object that you write is over the limit.

What to do

• Split large ConfigMap or Secret payloads into multiple smaller objects.
• Move large blobs (binaries, datasets, generated artifacts) out of Kubernetes and into object storage (for example, S3 or GCS), and reference them by URL.
• Trim large annotations. Annotations are for small metadata, not bulk data.
• For Custom Resources, review the schema. Large status or spec fields are a common cause.

Data store size quota (1 GiB)

The cluster’s data store (the etcd-compatible backend that holds all Kubernetes objects) has a total quota of 1 GiB across all Namespaces and resource types combined. When the data store reaches this quota, the API server stops accepting writes. It returns a NOSPACE condition until space is freed.

The quota applies to the total size of the data store. This size includes storage that deleted objects have not yet released. Background compaction reclaims space, so the reported size can lag behind the number of objects you currently have.

The most common symptom is failed kubectl apply/create calls with errors mentioning mvcc: database space exceeded or a NOSPACE alarm in the cluster status.

What to do

1. Delete objects you no longer need. The most common culprits are:
   • Old Job and completed Pod records that were never cleaned up.
   • Stale Event objects (these can accumulate in busy clusters).
   • Unused ConfigMap / Secret revisions left behind by Helm or rollout tooling.
   • Custom Resources from operators that don’t garbage-collect on their own.
2. Wait a few minutes. After deletion, the data store needs time to compact and reclaim space in the background. When the size drops below the quota, writes resume automatically. You do not need to take manual action on the cluster.
3. If writes still fail after deleting objects and waiting, open a ticket in the Cloudfleet support center. Some recovery paths need operator action on the underlying data store, which you cannot do yourself.

Avoiding the quota

The 1 GiB ceiling is large enough for most clusters. A typical cluster uses a small fraction of it, because Kubernetes objects are mostly small pieces of metadata. These patterns push clusters toward the ceiling faster than expected:

• Storing application data in ConfigMap or Secret objects instead of a real datastore.
• Running operators that create one Custom Resource per work item without cleanup.
• High-cardinality controllers that emit large numbers of Event objects.
• Security or policy scanners that write a report object per workload or per image.

If your workload needs more capacity, contact support before you hit the limit. Support can arrange a higher quota for clusters that legitimately need one. Recovery from a full data store is slower than planning ahead.