Network observability with Hubble

Cloudfleet Kubernetes Engine (CFKE) uses Cilium as its container network interface, and Cilium’s observability layer Hubble is already active on every node in your cluster. Each node continuously records network flows: which pods talk to each other, over which ports and protocols, and whether the traffic was forwarded or dropped.

What CFKE does not deploy by default are the two optional components that make this data easy to consume cluster-wide:

  • Hubble Relay, which aggregates the flow data of all nodes behind a single API endpoint.
  • Hubble UI, a web interface that visualizes live service dependencies and flows on top of Relay.

Most clusters do not need these components running permanently, so CFKE keeps them off to avoid spending your resources on idle workloads. This guide shows how to deploy both on demand with a single manifest, use them to inspect your network, and remove them again when you are done.

This is useful when you are debugging connectivity between services, verifying network policies, or simply want to understand what is flowing through your cluster.

How it works

The Hubble API is served by the Cilium agent on every node. Hubble Relay connects to each node’s Hubble endpoint, merges the flow streams, and exposes the aggregated API inside the cluster. The Hubble UI then reads from Relay and renders a live service map. The manifest below deploys both into the kube-system namespace and works without any additional setup.

Deploy Relay and UI

Save the following manifest as hubble-observability.yaml and apply it:

bash 
kubectl apply -f hubble-observability.yaml
yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hubble-relay
  namespace: kube-system
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hubble-ui
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hubble-relay-config
  namespace: kube-system
data:
  config.yaml: |
    cluster-name: default
    peer-service: "hubble-peer.kube-system.svc.cluster.local.:443"
    listen-address: :4245
    tls-hubble-client-cert-file: /var/lib/hubble-relay/tls/client.crt
    tls-hubble-client-key-file: /var/lib/hubble-relay/tls/client.key
    tls-hubble-server-ca-files: /var/lib/hubble-relay/tls/hubble-server-ca.crt
    disable-server-tls: true    
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hubble-ui-nginx
  namespace: kube-system
data:
  nginx.conf: |
    server {
        listen       8081;
        listen       [::]:8081;
        server_name  localhost;
        root /app;
        index index.html;
        client_max_body_size 1G;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;

            location /api {
                proxy_http_version 1.1;
                proxy_pass_request_headers on;
                proxy_pass http://127.0.0.1:8090;
            }
            location / {
                if ($http_user_agent ~* "kube-probe") { access_log off; }
                # double `/index.html` is required here
                try_files $uri $uri/ /index.html /index.html;
            }

            # Liveness probe
            location /healthz {
                access_log off;
                add_header Content-Type text/plain;
                return 200 'ok';
            }
        }
    }    
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hubble-ui
rules:
  - apiGroups:
      - ""
    resources:
      - componentstatuses
      - endpoints
      - namespaces
      - nodes
      - pods
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hubble-ui
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hubble-ui
subjects:
  - kind: ServiceAccount
    name: hubble-ui
    namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: hubble-relay
  namespace: kube-system
  labels:
    k8s-app: hubble-relay
spec:
  type: ClusterIP
  selector:
    k8s-app: hubble-relay
  ports:
    - protocol: TCP
      port: 80
      targetPort: grpc
---
apiVersion: v1
kind: Service
metadata:
  name: hubble-ui
  namespace: kube-system
  labels:
    k8s-app: hubble-ui
spec:
  type: ClusterIP
  selector:
    k8s-app: hubble-ui
  ports:
    - name: http
      port: 80
      targetPort: 8081
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hubble-relay
  namespace: kube-system
  labels:
    k8s-app: hubble-relay
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: hubble-relay
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: hubble-relay
    spec:
      securityContext:
        fsGroup: 65532
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: hubble-relay
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsGroup: 65532
            runAsNonRoot: true
            runAsUser: 65532
            seccompProfile:
              type: RuntimeDefault
          image: "quay.io/cilium/hubble-relay:v1.19.4"
          imagePullPolicy: IfNotPresent
          command:
            - hubble-relay
          args:
            - serve
          ports:
            - name: grpc
              containerPort: 4245
          resources:
            requests:
              cpu: 25m
              memory: 64Mi
          readinessProbe:
            grpc:
              port: 4222
            timeoutSeconds: 3
          livenessProbe:
            grpc:
              port: 4222
            timeoutSeconds: 10
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 12
          startupProbe:
            grpc:
              port: 4222
            initialDelaySeconds: 10
            failureThreshold: 20
            periodSeconds: 3
          volumeMounts:
            - name: config
              mountPath: /etc/hubble-relay
              readOnly: true
            - name: tls
              mountPath: /var/lib/hubble-relay/tls
              readOnly: true
          terminationMessagePolicy: FallbackToLogsOnError
      restartPolicy: Always
      serviceAccountName: hubble-relay
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 1
      nodeSelector:
        kubernetes.io/os: linux
      volumes:
        - name: config
          configMap:
            name: hubble-relay-config
            items:
              - key: config.yaml
                path: config.yaml
        - name: tls
          projected:
            # note: the leading zero means this number is in octal representation: do not remove it
            defaultMode: 0400
            sources:
              - secret:
                  name: hubble-server-certs
                  items:
                    - key: tls.crt
                      path: client.crt
                    - key: tls.key
                      path: client.key
                    - key: ca.crt
                      path: hubble-server-ca.crt
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hubble-ui
  namespace: kube-system
  labels:
    k8s-app: hubble-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: hubble-ui
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: hubble-ui
    spec:
      securityContext:
        fsGroup: 1001
        runAsGroup: 1001
        runAsUser: 1001
      serviceAccountName: hubble-ui
      automountServiceAccountToken: true
      containers:
        - name: frontend
          image: "quay.io/cilium/hubble-ui:v0.13.5"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8081
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          volumeMounts:
            - name: hubble-ui-nginx-conf
              mountPath: /etc/nginx/conf.d/default.conf
              subPath: nginx.conf
            - name: tmp-dir
              mountPath: /tmp
          terminationMessagePolicy: FallbackToLogsOnError
          securityContext:
            allowPrivilegeEscalation: false
        - name: backend
          image: "quay.io/cilium/hubble-ui-backend:v0.13.5"
          imagePullPolicy: IfNotPresent
          env:
            - name: EVENTS_SERVER_PORT
              value: "8090"
            - name: FLOWS_API_ADDR
              value: "hubble-relay.kube-system.svc.cluster.local:80"
          ports:
            - name: grpc
              containerPort: 8090
          resources:
            requests:
              cpu: 25m
              memory: 64Mi
          terminationMessagePolicy: FallbackToLogsOnError
          securityContext:
            allowPrivilegeEscalation: false
      nodeSelector:
        kubernetes.io/os: linux
      volumes:
        - configMap:
            defaultMode: 420
            name: hubble-ui-nginx
          name: hubble-ui-nginx-conf
        - name: tmp-dir
          emptyDir: {}

Within about half a minute both pods should be running:

bash 
kubectl get pods -n kube-system -l 'k8s-app in (hubble-relay,hubble-ui)'
NAME                            READY   STATUS    RESTARTS   AGE
hubble-relay-7c4b8d4989-44gtl   1/1     Running   0          30s
hubble-ui-6d7dfb644d-48fzm      2/2     Running   0          30s

The Relay logs should show a Connected line for every node in your cluster:

bash 
kubectl logs -n kube-system deploy/hubble-relay

Open the Hubble UI

The UI is intentionally not exposed to the internet. Access it through a port-forward:

bash 
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

Then open http://localhost:12000 in your browser. Select a namespace to see a live service map and the individual flows, including verdicts (forwarded or dropped), ports, and protocols.

Use the Hubble CLI

If you prefer the terminal, the Hubble CLI can query Relay through a second port-forward:

bash 
kubectl port-forward -n kube-system svc/hubble-relay 4245:80
bash 
hubble status --server localhost:4245
hubble observe --server localhost:4245 --follow

hubble observe supports rich filters, for example --namespace, --pod, --verdict DROPPED, or --protocol TCP. See the Hubble documentation for the full reference.

Clean up

When you are done, remove everything with a single command:

bash 
kubectl delete -f hubble-observability.yaml

The flow recording on the nodes is unaffected. You can redeploy Relay and the UI at any time by reapplying the manifest.

Key considerations

  • No authentication on the UI: The Hubble UI and the Relay API have no built-in authentication and expose network metadata for the entire cluster. Keep them on ClusterIP services and access them via kubectl port-forward as shown above. Do not expose them through a LoadBalancer service or an Ingress without adding an authentication layer in front.

  • Version compatibility: The hubble-relay image version should match the minor version of Cilium running in your cluster. The manifest above matches the Cilium version currently deployed by CFKE. You can check your cluster’s version with:

    bash 
    kubectl get daemonset cilium -n kube-system \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

  • Flow data is in memory only: Each node keeps its recent flows in a fixed-size ring buffer. Hubble shows you what is happening now and in the recent past; it is not a long-term storage or audit system.

Further reading

Previous
← Egress Gateways
On this page