Persistent volume issues

On CFKE, CSI drivers for persistent volumes are installed and owned by you (see the persistent volumes tutorial for Hetzner). This page covers the volume problems that reach support most often.

Volume stuck attaching or mounting (Hetzner)

Symptoms:

  • pod stuck in ContainerCreating with FailedAttachVolume: volume attachment is being deleted
  • FailedMount with a missing device path such as /dev/disk/by-id/scsi-0HC_Volume_VOLUME_ID
  • Multi-Attach error for volume when a pod moves between nodes

Fixes:

  1. Keep the CSI driver current. Older hcloud-csi releases contain attach and mount races fixed upstream (v2.21.2 or later fixes the “device needs to be ready on node publish” failure behind the missing /dev/disk/by-id path):

    bash 
    helm upgrade --install hcloud-csi hcloud/hcloud-csi -n kube-system

  2. Clear a stale VolumeAttachment. When a volume moves between nodes, a known upstream race can leave a VolumeAttachment object behind that blocks re-attachment. Deleting the stale object is non-destructive (it does not touch your data) and triggers a clean re-attach:

    bash 
    kubectl get volumeattachments
kubectl delete volumeattachment ATTACHMENT_NAME

Volumes and node placement

Block volumes are zonal: a volume created in one location can only be attached to nodes in the same location. The scheduler and the node auto-provisioner respect volume topology automatically, but keep it in mind when you constrain workloads: pinning a pod away from its volume’s region makes it unschedulable. If a StatefulSet must survive node consolidation without disruption, protect it with a PodDisruptionBudget and consider a conservative auto-provisioning profile.

AWS Fleets and the EBS CSI driver

Symptom: on auto-provisioned AWS nodes, the EBS/EFS CSI node plugin crash-loops with:

RegisterPlugin error ... detected topology value collision: driver reported
"topology.kubernetes.io/zone":"AZ_NAME" but existing label is
"topology.kubernetes.io/zone":"AZ_ID"

and PVCs stay Pending with no topology key found for node.

Cause: a known limitation: CFKE labels auto-provisioned AWS nodes with the availability zone ID (for example euw1-az2), while the AWS CSI drivers derive the zone name (for example eu-west-1a) from the instance metadata service, and the two collide.

Workarounds until the fix ships: use a provider-agnostic storage solution (such as Longhorn) on auto-provisioned nodes, or run AWS instances as self-managed nodes where you control the zone label. If this limitation affects you, contact support for current status.

Pods evicted with disk pressure

Symptom: pods are evicted with node was low on resource: ephemeral-storage and left in Failed or ContainerStatusUnknown; the node reports the DiskPressure condition.

What to check:

  1. Set ephemeral-storage requests and limits on workloads that write significant local data (logs, caches, temporary files), so the scheduler places them on nodes with enough disk and the kubelet can enforce a bound.
  2. Look for log-heavy containers without rotation and images with large writable layers.
  3. On auto-provisioned nodes, disk size follows the instance type; if your workloads legitimately need more local disk, steer them to larger instance types via resource requests or an instance family constraint.

Evicted pod records are not cleaned up automatically; after resolving the pressure, remove them with:

bash 
kubectl delete pods --field-selector=status.phase=Failed -A
Previous
← Kubernetes API server quotas
On this page