What does exit code 137 (OOMKilled) mean in Kubernetes?

Exit code 137 means a container’s main process was killed with SIGKILL, signal 9: by convention, a process killed by a signal exits with 128 plus the signal number, and 128 + 9 = 137. In Kubernetes the most common cause is the Linux out-of-memory (OOM) killer stopping a container that went over its memory limit, which Kubernetes reports as OOMKilled. A container can also get exit code 137 when the node itself runs out of memory or when the kubelet force-kills a container that did not stop in time.

Why the code is 137

The Bash manual states the rule: “When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status.” Container runtimes report signals the same way. SIGKILL cannot be caught or handled, so the process gets no chance to flush buffers, close connections or log an error. That is why a 137 often comes with logs that simply stop.

Exit code 143 (128 + 15) is the graceful counterpart: the process received SIGTERM and exited. A container that ignores SIGTERM gets SIGKILL after its grace period and then shows 137.

The four causes

CauseWhat kubectl describe pod showsWhat happens next
Container went over its memory limitLast State: Terminated, Reason: OOMKilled, Exit Code: 137The kubelet restarts the container on the same node
Node ran out of memory before the kubelet could reactUsually Reason: OOMKilled and Exit Code: 137, while usage was below the container’s own limitThe container restarts on the same node
Kubelet evicted the Pod under memory pressurePod Status: Failed, Reason: Evicted, message that the node was low on memoryThe Pod is not restarted; its controller creates a new one, possibly elsewhere
Liveness probe failed or shutdown took too longReason: Error, Exit Code: 137, events with Liveness probe failed or KillingThe container restarts; repeated kills lead to CrashLoopBackOff

Container memory limit

A memory limit is enforced by the kernel through the container’s cgroup. The Kubernetes documentation puts it this way: if a container “continues to consume memory beyond its limit, the Container is terminated”, and the kubelet restarts it if the restart policy allows. On nodes with cgroup v2, the kubelet by default makes an OOM kill stop every process in the container together, not only the largest one.

Node out of memory

When the whole node runs out of memory before the kubelet can free any, the kernel’s OOM killer chooses a victim across all containers. The kubelet steers that choice through oom_score_adj, based on the Pod’s quality of service class:

QoS classoom_score_adjKilled
BestEffort (no requests or limits)1000First
BurstableBetween 2 and 999, higher for Pods using more memory relative to their requestIn between
Guaranteed (limits equal requests)-997Last

Node-pressure eviction

The kubelet watches free memory on the node. By default on Linux it starts evicting Pods when available memory drops below 100 MiB. It evicts Pods that use more than their requests first, and with this hard threshold it stops them without a grace period. An evicted Pod is not restarted in place; its phase is Failed and kubectl get pods shows Evicted.

How to tell the causes apart

  1. Read the last termination of the container:

    bash
    kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

    reason: OOMKilled points to memory. reason: Error with code 137 points to a forced kill, usually a probe.

  2. Compare the container’s memory limit with its usage before the kill. kubectl top pod <pod> --containers shows current usage; your metrics system shows the history.

  3. Check the node for memory pressure:

    bash
    kubectl describe node <node>

    A MemoryPressure condition or eviction events point to the node, not the container.

  4. If you have node access, the kernel log names the trigger. Memory cgroup out of memory means a container hit its own limit; Out of memory without the cgroup prefix means the node itself ran out.

How to fix it

  • Set the memory limit from measured peak usage. Watch memory over a full load cycle, then set the limit above the peak with headroom. Setting the request equal to the limit makes the scheduler reserve that memory and gives the Pod the Guaranteed class. Kubernetes requests and limits covers the trade-offs.
  • Make the runtime respect the limit. Garbage-collected runtimes do not always keep their heap below the container limit. Set the heap size explicitly: -XX:MaxRAMPercentage for the JVM, --max-old-space-size for Node.js, GOMEMLIMIT for Go.
  • Profile before raising limits. If usage climbs until the kill no matter how high the limit is, the application leaks memory. Take heap profiles or dumps and compare them over time.
  • Stop overcommitting nodes. Node OOM kills and evictions happen when the sum of actual usage exceeds the node’s memory, which is likely when requests are far below real usage. Raise requests to realistic values, and reserve memory for system daemons with the kubelet’s kube-reserved and system-reserved settings.
  • Tune probes. If the reason is Error, not OOMKilled, give slow starters a startupProbe and make sure the application exits on SIGTERM within its grace period.
  • Use recommendations. The Vertical Pod Autoscaler can suggest requests from observed usage; see how Kubernetes autoscaling works.

Memory on Cloudfleet

In Cloudfleet Kubernetes Engine (CFKE), node auto-provisioning sizes new nodes from Pod requests, so realistic memory requests also keep nodes from being packed beyond what their workloads use.

On this page