Network policies

Network policies act as firewall rules for Pods. By default, every Pod in a Kubernetes cluster can reach every other Pod and any external address. With network policies, you define which traffic to allow. Cilium drops everything else.

CFKE uses Cilium to enforce network policies on every node, in all clouds and on self-managed nodes. You do not need to enable anything. A policy takes effect within seconds after you apply it. To decide which policy type fits your goal, start with Network security.

This page covers the most common patterns. For a description of every field, see the upstream references:

Kubernetes NetworkPolicy

The standard NetworkPolicy resource applies to one namespace and selects Pods by labels. It works the same on CFKE as on any other conformant Kubernetes cluster. Your existing manifests and Helm charts work without changes.

Default-deny in a namespace

Start every protected namespace with a default-deny policy. The policy selects all Pods in the namespace and allows nothing. Only traffic that other policies explicitly allow gets through:

yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

When egress is denied, Pods cannot resolve DNS names. To keep service discovery working, allow DNS traffic to CoreDNS:

yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: coredns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Allow traffic between applications

The following policies allow Pods with the label app: frontend to reach Pods with the label app: api on port 8080. The default-deny policy also blocks egress, so the frontend needs a matching egress rule:

yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-allow-api
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080

To allow traffic from another namespace, add a namespaceSelector next to the podSelector. For example, select kubernetes.io/metadata.name: monitoring to let Prometheus scrape your Pods.

CiliumNetworkPolicy

CiliumNetworkPolicy is a Cilium resource that applies to one namespace. It extends NetworkPolicy with Layer 7 rules, egress by domain name, predefined entities, and explicit deny rules. Use it when the standard resource cannot express what you need. You can use both resource types in the same namespace.

Layer 7 HTTP rules

Layer 7 rules inspect requests and allow only specific HTTP methods and paths. The following policy lets the frontend read from the API, but not write to it:

yaml 
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-read-only-for-frontend
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: "/api/.*"

Requests that do not match get an HTTP 403 Forbidden response, not a dropped connection. Layer 7 inspection sends the traffic through a proxy on the node, which adds some latency. Apply it only to the services that need it. Layer 7 rules work only on unencrypted HTTP. For TLS traffic between services, use a service mesh such as Istio.

Cilium enforces Layer 7 rules only when no other policy allows the same traffic at Layer 4. Allow rules are additive. For example, the api-allow-frontend policy above already allows the frontend to reach port 8080. With that policy in place, every HTTP request passes and the Layer 7 rules have no effect. Replace the Layer 4 rule for that port with the CiliumNetworkPolicy. Do not add the CiliumNetworkPolicy next to it.

Egress by domain name

External services rarely have stable IP addresses. A toFQDNs rule allows egress by domain name. Cilium updates the allowed IP addresses from the DNS responses that the Pod receives:

yaml 
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-egress
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: payments
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: coredns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toFQDNs:
        - matchName: "api.stripe.com"
        - matchPattern: "*.s3.eu-central-1.amazonaws.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

The first rule is required. Cilium learns the IP addresses of a domain from the DNS queries it inspects. The dns rule sends DNS traffic through Cilium’s DNS proxy. Without it, toFQDNs rules never match.

Keep the following in mind:

  • matchPattern: "*.example.com" matches subdomains, but not example.com itself. Add a matchName for the apex domain if you need it.
  • Services behind shared IP addresses, such as CDNs, can serve other domains on the same IP. Domain-based rules limit what a Pod can resolve and connect to, but they do not replace TLS verification.
  • If the external service allow-lists source IP addresses, combine this policy with an egress gateway. Outbound traffic then leaves from a fixed IP address.

Entities

Entities are predefined identities. You reference them without labels or CIDRs. The most common entities are:

  • world: everything outside the cluster.
  • kube-apiserver: the Kubernetes API server.

The following rules let a Pod reach the internet and the API server:

yaml 
egress:
  - toEntities:
      - world
      - kube-apiserver

The kube-apiserver entity alone is enough for clients that use the in-cluster kubernetes.default Service. For the full list of entities, see the Cilium documentation.

Cluster-wide policies

CiliumClusterwideNetworkPolicy uses the same syntax as CiliumNetworkPolicy, but applies to Pods in every namespace. Platform and security teams use it for guardrails that application teams cannot override. Only users with cluster-level permissions can create cluster-scoped resources. You can give teams full control over the policies in their own namespaces and still keep these guardrails out of their reach.

Block cloud metadata endpoints

Cloud metadata endpoints can expose instance credentials to a compromised Pod. The following policy blocks access to them from all Pods, even if a namespace policy allows it:

yaml 
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-cloud-metadata
spec:
  endpointSelector: {}
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
    - toCIDR:
        - 169.254.169.254/32

The enableDefaultDeny field is important here. Without it, the policy switches every Pod it selects to default-deny, and Cilium blocks all other traffic. When you set it to false, the policy only adds the deny rule. All other traffic is unchanged.

Isolate a sensitive namespace

The following policy blocks all traffic from Pods in other namespaces to the payments namespace. Application teams can still allow traffic within the namespace and from outside the cluster:

yaml 
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: isolate-payments
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: payments
  enableDefaultDeny:
    egress: false
    ingress: false
  ingressDeny:
    - fromEndpoints:
        - matchExpressions:
            - key: k8s:io.kubernetes.pod.namespace
              operator: Exists
            - key: k8s:io.kubernetes.pod.namespace
              operator: NotIn
              values:
                - payments

The Exists expression limits the rule to Pods. Without it, NotIn also matches identities without a namespace label, such as nodes and clients outside the cluster. If an ingress controller or monitoring stack in another namespace must reach these Pods, add its namespace to the NotIn list.

Example: isolate tenants across data centers

A single CFKE cluster can span many sites. For example, each of your customers provides servers in their own data center, and you join these servers to one cluster as self-managed nodes. All nodes share one encrypted overlay network, so by default every Pod can reach every other Pod, across tenants and sites. The following setup keeps each tenant’s workloads on the tenant’s own servers and blocks all traffic between tenants.

Step 1: Dedicate nodes to each tenant

Register each tenant’s servers with a tenant label and a taint. The taint keeps the workloads of other tenants off these servers:

bash 
cloudfleet clusters add-self-managed-node CLUSTER_ID \
  --host HOST_IP \
  --region DATACENTER_REGION \
  --zone DATACENTER_ZONE \
  --label tenant=acme \
  --taint tenant=acme:NoSchedule

With Terraform, use the node_labels and node_taints attributes of the self-managed node resources.

Step 2: Label the tenant’s namespaces

Run each tenant’s workloads in namespaces with the same tenant label. A tenant can have several namespaces, for example for production and staging:

yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: acme-prod
  labels:
    tenant: acme

In the Pod templates of the tenant’s workloads, add a node selector and a toleration, so that the Pods run only on the tenant’s servers:

yaml 
nodeSelector:
  tenant: acme
tolerations:
  - key: tenant
    value: acme
    effect: NoSchedule

Step 3: Block traffic between tenants

Create one CiliumClusterwideNetworkPolicy for each tenant. The policy selects all Pods in namespaces with the tenant’s label. It blocks traffic to and from Pods in namespaces that belong to a different tenant:

yaml 
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: isolate-tenant-acme
spec:
  endpointSelector:
    matchLabels:
      k8s:io.cilium.k8s.namespace.labels.tenant: acme
  enableDefaultDeny:
    ingress: false
    egress: false
  ingressDeny:
    - fromEndpoints:
        - matchExpressions:
            - key: k8s:io.cilium.k8s.namespace.labels.tenant
              operator: Exists
            - key: k8s:io.cilium.k8s.namespace.labels.tenant
              operator: NotIn
              values:
                - acme
    - fromEntities:
        - remote-node
  egressDeny:
    - toEndpoints:
        - matchExpressions:
            - key: k8s:io.cilium.k8s.namespace.labels.tenant
              operator: Exists
            - key: k8s:io.cilium.k8s.namespace.labels.tenant
              operator: NotIn
              values:
                - acme

Generate one policy per tenant from a template, for example with Helm or Terraform. With these policies in place:

  • Pods of different tenants cannot connect to each other, in either direction.
  • Pods of the same tenant can connect across namespaces and across the tenant’s servers.
  • Namespaces without a tenant label, such as kube-system or a shared monitoring namespace, can still reach all tenants. Cluster DNS, the Kubernetes API, and the internet stay reachable.
  • The remote-node rule blocks traffic from other nodes’ host network, such as a hostNetwork Pod on another tenant’s server. Traffic from load balancers and readiness probes is not affected.

Keep the following in mind:

  • Protect the tenant label. Any namespace with the label tenant: acme belongs to the tenant acme. Only platform administrators must be able to create namespaces and set this label. Do not give tenants permission to create or label namespaces.
  • Label every tenant namespace. A namespace without a tenant label is treated as shared and can reach every tenant.
  • Combine with Kubernetes RBAC. Network policies isolate traffic. To isolate access to the Kubernetes API, give each tenant RBAC permissions only in its own namespaces.
  • Use separate clusters for hard isolation. Tenants in one cluster share the control plane and the cluster-wide resources. If tenants must not share anything, give each tenant its own cluster.

CFKE considerations

  • Pods behind load balancers: traffic from load balancers comes from outside the cluster. In a default-deny namespace, allow it with a CiliumNetworkPolicy ingress rule. Use fromEntities: [world] on the container port.
  • ipBlock does not match Pods: Cilium applies ipBlock and toCIDR rules only to addresses outside the cluster. For traffic inside the cluster, use Pod and namespace selectors.
  • Host network Pods: policies do not apply to Pods with hostNetwork: true. These Pods share the network namespace of the node.
  • Node encryption is independent: WireGuard® encryption between nodes stays active, whatever your policies are. Policies decide whether to allow traffic. Encryption protects the traffic in transit. See Networking architecture.

Verify and troubleshoot policies

List the policies that apply in your cluster:

bash 
kubectl get networkpolicies,ciliumnetworkpolicies --all-namespaces
kubectl get ciliumclusterwidenetworkpolicies

To see how your policies affect live traffic:

  1. Deploy Hubble Relay as described in Network observability with Hubble.
  2. Watch for dropped connections in a namespace:
bash 
hubble observe --server localhost:4245 --namespace shop --verdict DROPPED --follow

Each dropped flow shows the source, the destination, the port, and the reason, such as Policy denied. To see the policy decisions for allowed and denied connections, filter by event type:

bash 
hubble observe --server localhost:4245 --namespace shop --type policy-verdict --follow

To roll out a new default-deny policy safely:

  1. Apply the policy in a staging namespace or cluster.
  2. Use the application and watch hubble observe --verdict DROPPED for unexpected drops.
  3. Add allow rules for legitimate traffic until no unexpected drops remain.
  4. Apply the same policies in production.

If you test with a temporary debug Pod, give it the labels that your policies select on. Your allow rules do not match an unlabelled Pod, so Cilium drops its traffic. This is easy to mistake for a platform issue. For more common issues, see Network connectivity issues.

Previous
← Network security
Next
Public cloud load balancing →
On this page