Storage overview

Cloudfleet Kubernetes Engine (CFKE) runs the block storage of each cloud in your Fleet: Amazon EBS on AWS, Persistent Disk on GCP, and Volumes on Hetzner Cloud. You install the cloud provider’s Container Storage Interface (CSI) driver, and CFKE takes care of the rest. Pods land in the zone of their volume, and volumes follow their Pods when nodes are replaced.

Choose a storage type

• Cloud block volumes: Amazon EBS, GCP Persistent Disk, and Hetzner Cloud Volumes attach to nodes in the zone where they were created.
• Local disks: local disks give fast storage for scratch data and for apps that replicate data themselves. The data stays on one node.
• Replicated block storage: Longhorn or Rook-Ceph replicate volumes across nodes, so a volume survives the loss of a node.
• Shared file storage: NFS, Amazon EFS, and GCP Filestore provide ReadWriteMany volumes through their CSI drivers.
• Ephemeral volumes: emptyDir and generic ephemeral volumes exist only as long as the Pod. Set ephemeral-storage requests, so the scheduler accounts for the disk space.

Keyless by design

The CSI drivers on AWS and GCP need no access keys or service account keys. Every Pod in a CFKE cluster carries an OIDC token issued by the cluster. AWS IAM and GCP Workload Identity Federation trust this token directly, so the driver exchanges it for short-lived cloud credentials. You keep no long-lived secret in the cluster, nothing to rotate, and nothing to leak. Accessing cloud APIs securely explains the mechanism.

Hetzner Cloud does not support identity federation. The Hetzner CSI driver reuses the API token that your Fleet already stores in the cluster, so you do not create a second credential.

Install the driver for each cloud where your Pods need persistent volumes. The drivers run side by side in one cluster.

How it works

A CSI driver has two parts:

• The controller creates, attaches, resizes, and deletes volumes through the cloud API.
• The node plugin mounts volumes on the node. It runs as a DaemonSet.

Both run on the nodes of their own cloud, selected with the cfke.io/provider label.

Block volumes are zonal: a volume attaches only to nodes in the zone, or the Hetzner Cloud location, where it was created. CFKE labels every auto-provisioned node with the topology key of its cloud’s CSI driver, alongside the standard topology.kubernetes.io/zone and topology.kubernetes.io/region labels:

When a Pod needs a volume in a zone without a free node, the node auto-provisioner launches a node in exactly that zone. Pods therefore keep their data when they move: during node consolidation, when a node fails, and when you replace nodes.

Storage classes

Each driver comes with a storage class for its cloud. Use volumeBindingMode: WaitForFirstConsumer in every storage class. The volume is then created in the zone of the node where the scheduler places the Pod, instead of in a zone that the Pod cannot reach.

In a Fleet with several clouds, a volume works only on nodes of its own cloud. Set storageClassName on every claim, because a claim without it gets the cluster’s default class. Select the cloud of the Pods that use the volume with a node selector:

spec:
  nodeSelector:
    cfke.io/provider: aws # or gcp, hetzner

Volume expansion

All three drivers support online expansion. Increase spec.resources.requests.storage of the PersistentVolumeClaim, and the file system grows while the Pod keeps running. The storage class must set allowVolumeExpansion: true.

Self-managed nodes

On self-managed nodes, use the CSI driver of your storage system, or local disks for node-local storage. Local storage ties each volume to one node.

Troubleshooting

See persistent volume issues for volumes stuck attaching or mounting, and for Pods that cannot reach their volume.