Deploy a Prometheus monitoring stack on Kubernetes

“Install Prometheus” sounds like one decision. It is closer to four: how metrics get collected, where they are stored, what draws the graphs, and what wakes someone up. The Prometheus ecosystem offers a different project for each layer, which is why the same question gets a dozen different answers.

This guide starts by narrowing that down. It walks through the realistic options, gives you a way to pick between them, and then deploys the one that fits most teams: kube-prometheus-stack, a single Helm chart that installs Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics, already wired together with working dashboards and alert rules.

The deployment targets Cloudfleet Kubernetes Engine (CFKE) with nodes on Hetzner Cloud, which is a sensible pairing for monitoring: Prometheus is memory-hungry and stores everything on disk, and both are cheap there. Several defaults in that chart assume you run your own control plane, which on a managed platform produces alerts that fire forever about components you do not operate. Those are called out and fixed as they come up.

Choosing how to deploy Prometheus

It helps to separate the stack into layers, because most of the competing projects only replace one of them.

A single Prometheus does collection, storage and rule evaluation on its own. Everything else in that table exists because one of those jobs stops working well at a certain scale or retention.

The realistic options

kube-prometheus-stack. One Helm chart, all four layers, sensible defaults. It installs the Prometheus Operator, so Prometheus configuration becomes Kubernetes resources: a ServiceMonitor tells Prometheus to scrape a service, a PrometheusRule defines alerts. You never hand-edit prometheus.yml. It ships the community dashboard and alert rule sets that most people would otherwise copy in by hand. This is the right default and what the rest of this guide deploys.

Prometheus Operator on its own. The same operator without the bundled Grafana, dashboards and rules. Choose this when you already run Grafana elsewhere, or when your platform team wants to curate the rules rather than start from the community set.

The plain prometheus Helm chart. Prometheus with a config file and no operator. Fewer moving parts and no CRDs, but every new scrape target means editing the config. Reasonable for a small static cluster, painful once teams start deploying their own services.

kube-prometheus (jsonnet). The upstream project the Helm chart is generated from. Maximum control, and you need to be comfortable with jsonnet. Choose it when you are templating many clusters with real differences between them.

VictoriaMetrics. A drop-in replacement for Prometheus storage that speaks PromQL and typically uses noticeably less memory and disk for the same series. Its operator mirrors the Prometheus Operator’s CRDs, so ServiceMonitor resources carry over. Worth it when a single Prometheus is straining on memory or you are keeping months of data on one cluster.

Thanos or Mimir. Long-term storage and a global query view across clusters. They sit behind Prometheus rather than replacing it: Prometheus keeps scraping, and ships blocks to object storage. Add one when you need multi-cluster queries or retention measured in quarters, not before.

Grafana Alloy or the OpenTelemetry Collector. Collection agents that scrape Prometheus endpoints and forward them somewhere else via remote_write. Use these when metrics, logs and traces should share one pipeline, or when storage lives in a managed backend.

Managed backends. Grafana Cloud, Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus. You still run a collector in-cluster; they take over storage, availability and retention. The trade is operational load for cost and for your metrics leaving your infrastructure.

How to decide

Work down this list and stop at the first one that matches.

1. You want dashboards and alerts working today on one cluster. Use kube-prometheus-stack. This covers most teams, and nothing below is cheaper to operate.
2. You already run Grafana, or you curate your own alert rules. Use the Prometheus Operator alone and point your existing Grafana at it.
3. You need to query several clusters together, or retain data for many months. Keep Prometheus for scraping and add Thanos or Mimir behind it, with object storage. Do not try to solve this by growing one Prometheus.
4. A single Prometheus is running out of memory on one cluster. Move storage to VictoriaMetrics before you reach for a distributed system. Your ServiceMonitor resources come with you.
5. Metrics, logs and traces should share a pipeline, or you would rather not run storage at all. Use Grafana Alloy or the OpenTelemetry Collector with remote_write to a managed backend.

Two things people get wrong in both directions. Reaching for Thanos or Mimir on day one buys a large operational surface to solve a problem you do not have yet; a single Prometheus handles a normal cluster comfortably. And staying on a hand-edited config file past the point where several teams deploy their own services turns every new service into a pull request against the monitoring config, which is exactly what the operator’s CRDs exist to avoid.

Prerequisites

• A running CFKE cluster with a fleet attached. See the getting started guide if you need one.
• kubectl and helm on your local machine.
• A storage class that can provision block volumes. Prometheus, Alertmanager and Grafana all want persistent storage. On Hetzner, follow persistent volumes with Cloudfleet on Hetzner to install the Hetzner Cloud CSI driver, which gives you a default hcloud-volumes class.

This guide uses hcloud-volumes and Hetzner nodes. On AWS, Google Cloud, or self-managed nodes, substitute your storage class name and the cfke.io/provider value in the node selectors.

What the chart installs

Before configuring it, it is worth knowing what the chart actually deploys, because you will disable some of it.

It also creates ServiceMonitor resources for control plane components: kube-scheduler, kube-controller-manager, etcd and kube-proxy. Those matter in the next section.

Step 1: Write values for a managed control plane

The chart assumes a self-managed cluster where you can scrape the control plane. On CFKE the control plane is managed for you and its components are not exposed as endpoints in your cluster. Left at defaults, those ServiceMonitor resources match nothing, so they produce zero targets rather than failing ones, and the accompanying alert rules go off:

pending  KubeControllerManagerDown
pending  KubeSchedulerDown
pending  KubeProxyDown
pending  TargetDown

That is the failure mode worth understanding. Nothing appears broken on the targets page, because a job with no targets is simply absent from it, but after the alerts’ for duration elapses your first monitoring install starts paging about a control plane you do not run. Turn those components off explicitly.

CoreDNS needs the opposite treatment. It runs in your cluster and is worth monitoring, but the chart’s service selector looks for the label k8s-app: kube-dns, while CFKE labels CoreDNS pods k8s-app: coredns. The selector matches nothing and DNS metrics go missing silently. Point it at the right label.

Create kube-prometheus-stack-values.yaml:

# CFKE runs the control plane, so there are no kube-scheduler,
# kube-controller-manager or etcd endpoints in the cluster to scrape.
# Left enabled, these produce zero targets and their alert rules fire forever.
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
# Cilium provides kube-proxy's functionality on CFKE, so there is no
# kube-proxy DaemonSet to scrape either.
kubeProxy:
  enabled: false

# CFKE labels CoreDNS pods k8s-app=coredns. The chart's default selector
# looks for kube-dns, matches nothing, and yields no DNS metrics.
coreDns:
  enabled: true
  service:
    selector:
      k8s-app: coredns

prometheus:
  prometheusSpec:
    retention: 15d
    nodeSelector:
      cfke.io/provider: hetzner
    resources:
      requests:
        cpu: 200m
        memory: 2Gi
      limits:
        memory: 2Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: hcloud-volumes
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

alertmanager:
  alertmanagerSpec:
    nodeSelector:
      cfke.io/provider: hetzner
    resources:
      requests:
        cpu: 10m
        memory: 100Mi
      limits:
        memory: 100Mi
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: hcloud-volumes
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

grafana:
  nodeSelector:
    cfke.io/provider: hetzner
  # A block volume attaches to one node at a time, so a rolling update
  # deadlocks: the new pod cannot mount the volume the old pod still holds.
  deploymentStrategy:
    type: Recreate
  resources:
    requests:
      cpu: 50m
      memory: 256Mi
    limits:
      memory: 256Mi
  sidecar:
    resources:
      requests:
        cpu: 10m
        memory: 128Mi
      limits:
        memory: 128Mi
  persistence:
    enabled: true
    storageClassName: hcloud-volumes
    size: 10Gi

prometheusOperator:
  nodeSelector:
    cfke.io/provider: hetzner
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      memory: 128Mi
  prometheusConfigReloader:
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        memory: 64Mi

kube-state-metrics:
  nodeSelector:
    cfke.io/provider: hetzner
  resources:
    requests:
      cpu: 20m
      memory: 128Mi
    limits:
      memory: 128Mi

prometheus-node-exporter:
  resources:
    requests:
      cpu: 20m
      memory: 64Mi
    limits:
      memory: 64Mi

Five things in that file are worth explaining.

Resource requests on every component. CFKE sizes and provisions nodes from pod resource requests, and this chart ships almost none. Installing it unchanged produces a warning per component:

Warning: Validation failed for ValidatingAdmissionPolicy 'resource-requests-are-not-set':
Containers without requests: node-exporter.

This is a warning, not a rejection. The pods schedule and the stack runs. But requests are the only signal the auto-provisioner has for sizing nodes, so leaving them out means it works from bad information: nodes can end up too small for what lands on them, and scale-up decisions get worse as the cluster fills. CFKE runs a handful of these policies, most of them advisory in the same way. Setting requests costs a few lines and is the difference between the provisioner guessing and knowing.

The node selectors only matter on mixed clusters. cfke.io/provider: hetzner pins these components to Hetzner nodes, which is necessary because a Hetzner Cloud volume can only attach to a Hetzner server. If your fleet uses a single provider, every node already satisfies that selector and you can leave it out: CFKE will place the pods on whatever it provisions. What you do need in a single-provider cluster is a storage class that works there, which is the real prerequisite. Add the selectors back as soon as the cluster spans two providers or includes self-managed nodes, so that pods with volumes land where their volumes can follow.

Memory limits but no CPU limits. A memory limit protects the node from a runaway container. A CPU limit only introduces throttling when the node has capacity to spare. The one number to watch here is Prometheus at 2Gi, which is comfortable for a small cluster and the first thing to raise as your series count grows.

Grafana’s sidecars need real memory. The two k8s-sidecar containers that load dashboards and datasources are OOM-killed at 64Mi. 128Mi is enough. If Grafana comes up 1/3 with the sidecars restarting, this is why.

Grafana needs Recreate. With persistence enabled on a ReadWriteOnce volume, the default rolling update cannot work: Kubernetes tries to start the new pod before removing the old one, and the volume is still attached to the old pod’s node. You get Multi-Attach error for volume and the rollout hangs until you delete the old ReplicaSet by hand. Set this before the first install. Changing it later is awkward, because the existing Deployment keeps its spec.strategy.rollingUpdate field and the API rejects the update until that field is cleared.

Step 2: Install the stack

Add the chart repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

Install:

helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  --version 88.5.3 \
  -n monitoring --create-namespace \
  --values kube-prometheus-stack-values.yaml

Pin the chart version. This chart changes quickly, and an unpinned install will not match this guide for long. Version 88.5.3 ships Prometheus v3.14.0, Grafana 13.2.0, Alertmanager v0.34.0 and Prometheus Operator v0.93.1.

On a cluster with spare capacity the pods start in a couple of minutes. On an empty one, expect longer while CFKE provisions nodes for them and attaches volumes:

kubectl get pods -n monitoring

NAME                                                   READY   STATUS    RESTARTS
alertmanager-kps-kube-prometheus-stack-alertmanager-0  2/2     Running   0
kps-grafana-665955c9dc-jn97q                           3/3     Running   0
kps-kube-prometheus-stack-operator-54687754c7-wpr5g    1/1     Running   0
kps-kube-state-metrics-6bb5f79f89-xf2hv                1/1     Running   0
kps-prometheus-node-exporter-2lwrv                     1/1     Running   0
prometheus-kps-kube-prometheus-stack-prometheus-0      2/2     Running   0

There is one node-exporter pod per node, so that list grows with the cluster.

Step 3: Verify what is actually being scraped

A monitoring stack that installed cleanly is not the same as a monitoring stack that is collecting anything. Check the targets.

kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090

Open http://localhost:9090/targets. Every job should be up, and there should be no jobs with zero targets:

apiserver             1/1
coredns               6/6
kube-state-metrics    1/1
kubelet              18/18
node-exporter         6/6

The number of kubelet targets is three per node, because Prometheus scrapes the kubelet’s own metrics, cAdvisor container metrics and probe metrics separately.

If coredns shows 0 targets, the service selector did not match. Confirm the label CFKE actually uses:

kubectl get pods -n kube-system -l k8s-app=coredns

Then check the alerts, at http://localhost:9090/alerts on the same port-forward. Look for anything in the PENDING state: a rule whose condition is true but whose for duration has not elapsed yet. If KubeSchedulerDown, KubeControllerManagerDown or KubeProxyDown appear there, the control plane components in step 1 are still enabled and those alerts will start firing shortly.

Check this in Prometheus rather than in Alertmanager. Alertmanager only ever receives alerts that have already transitioned to firing, so a freshly installed stack whose control plane alerts are still counting down looks completely clean there. Alertmanager is the right place to confirm routing, not to catch this:

kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-alertmanager 9093:9093

At http://localhost:9093 a healthy stack shows exactly one alert, Watchdog. That one is deliberate: it always fires, so its absence is the signal that your alerting pipeline has broken.

Step 4: Open Grafana

Get the generated admin password:

kubectl get secret -n monitoring kps-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d ; echo

Forward the port and log in as admin:

kubectl port-forward -n monitoring svc/kps-grafana 3000:80

At http://localhost:3000, the Prometheus datasource is already configured and the community Kubernetes dashboards are loaded. Start with Kubernetes / Compute Resources / Cluster for a whole-cluster view, and Node Exporter / Nodes for per-node detail. The CoreDNS dashboard is a quick way to confirm the selector fix from step 1 worked, since it stays blank if those targets never appeared.

You will not find dashboards for the scheduler, controller manager or etcd. The chart only installs those alongside the components themselves, so disabling them in step 1 leaves them out rather than leaving them empty. That is the correct outcome on a managed control plane: CFKE operates those components and monitors them itself.

For anything beyond a quick look, expose Grafana properly rather than port-forwarding. See expose HTTP applications with NGINX Ingress for an Ingress with automatic TLS, and put authentication in front of it.

Step 5: Monitor your own applications

The reason to run the operator is that adding a service to monitoring becomes a Kubernetes resource rather than a config change. Any service exposing Prometheus metrics needs a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

The selector matches your Kubernetes Service, and port refers to the service’s port by name, not by number. Prometheus picks it up within a scrape interval or two.

Alert rules work the same way:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app
  namespace: my-app
spec:
  groups:
    - name: my-app
      rules:
        - alert: MyAppHighErrorRate
          expr: |
            sum(rate(http_requests_total{job="my-app",status=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="my-app"}[5m])) > 0.05            
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "my-app is returning 5xx for more than 5% of requests"

By default this chart’s Prometheus picks up ServiceMonitor and PrometheusRule resources from every namespace. If you narrow that with serviceMonitorNamespaceSelector, remember to include the namespaces your teams deploy into.

Step 6: Route alerts somewhere

Prometheus evaluates the rules and hands firing alerts to Alertmanager, which groups and deduplicates them. Out of the box the chart routes everything to a receiver named null, so nothing leaves the cluster. Replace that route with a real one:

alertmanager:
  config:
    route:
      group_by: ["alertname", "namespace"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: "slack"
      routes:
        - matchers:
            - alertname = "Watchdog"
          receiver: "null"
    receivers:
      - name: "null"
      - name: "slack"
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/slack/url
            channel: "#alerts"
            send_resolved: true
  alertmanagerSpec:
    secrets:
      - slack-webhook

Create the referenced secret before upgrading:

kubectl create secret generic slack-webhook -n monitoring \
  --from-literal=url='https://hooks.slack.com/services/...'

Mounting the webhook from a secret keeps it out of your values file and out of version control. The explicit null route for Watchdog stops the heartbeat alert from reaching your channel, while leaving it available for a dead man’s switch service to watch.

Cleaning up

helm uninstall kps -n monitoring

Helm does not delete PersistentVolumeClaims created from volume claim templates, so your Prometheus, Alertmanager and Grafana volumes survive the uninstall. Remove them explicitly:

kubectl delete pvc -n monitoring --all
kubectl delete namespace monitoring

Helm also leaves the Prometheus Operator CRDs in place, which is deliberate: deleting them would delete every ServiceMonitor and PrometheusRule in the cluster along with them. Remove them only if you are done with the operator for good:

kubectl get crd -o name | grep monitoring.coreos.com | xargs kubectl delete

Where this stops working

The setup above is one Prometheus with local storage, which is the right shape for a single cluster. Two limits are worth knowing before you hit them.

Memory grows with active series, not with cluster size. A few thousand pods is unremarkable; one badly labelled application that puts a user ID or a timestamp in a metric label can multiply your series count overnight. If Prometheus starts being OOM-killed, look for the cardinality before you raise the limit. The TSDB Status page in Prometheus, under Status, ranks metrics by series count.

Retention is bounded by one disk. retention: 15d on a 20Gi volume is a starting point. You can grow the volume, but when you need months of history or a query across several clusters, that is the point to add Thanos or Mimir behind Prometheus, or to remote_write to a managed backend, rather than to keep scaling one instance.

Neither is a reason to build a distributed metrics system on day one. Start here, and let the metrics tell you when you have outgrown it.

Next steps

• Long-term storage. Thanos and Grafana Mimir both add object storage and a global query view across clusters.
• Logs alongside metrics. Loki uses the same label model as Prometheus and the same Grafana instance, so alerts and logs share a vocabulary.
• Lower-overhead storage. The VictoriaMetrics operator reads the same ServiceMonitor resources, which makes it a comparatively cheap migration if memory becomes the constraint.
• Fewer false pages. The community alert rules are a starting point, not a finished policy. Review them, delete what does not apply to a managed control plane, and tune the thresholds that page you at night.