What does exit code 137 (OOMKilled) mean in Kubernetes?
Exit code 137 means a container’s main process was killed with SIGKILL, signal 9: by convention, a process killed by a signal exits with 128 plus the signal number, and 128 + 9 = 137. In Kubernetes the most common cause is the Linux out-of-memory (OOM) killer stopping a container that went over its memory limit, which Kubernetes reports as OOMKilled. A container can also get exit code 137 when the node itself runs out of memory or when the kubelet force-kills a container that did not stop in time.
Why the code is 137
The Bash manual states the rule: “When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status.” Container runtimes report signals the same way. SIGKILL cannot be caught or handled, so the process gets no chance to flush buffers, close connections or log an error. That is why a 137 often comes with logs that simply stop.
Exit code 143 (128 + 15) is the graceful counterpart: the process received SIGTERM and exited. A container that ignores SIGTERM gets SIGKILL after its grace period and then shows 137.
The four causes
| Cause | What kubectl describe pod shows | What happens next |
|---|---|---|
| Container went over its memory limit | Last State: Terminated, Reason: OOMKilled, Exit Code: 137 | The kubelet restarts the container on the same node |
| Node ran out of memory before the kubelet could react | Usually Reason: OOMKilled and Exit Code: 137, while usage was below the container’s own limit | The container restarts on the same node |
| Kubelet evicted the Pod under memory pressure | Pod Status: Failed, Reason: Evicted, message that the node was low on memory | The Pod is not restarted; its controller creates a new one, possibly elsewhere |
| Liveness probe failed or shutdown took too long | Reason: Error, Exit Code: 137, events with Liveness probe failed or Killing | The container restarts; repeated kills lead to CrashLoopBackOff |
Container memory limit
A memory limit is enforced by the kernel through the container’s cgroup. The Kubernetes documentation puts it this way: if a container “continues to consume memory beyond its limit, the Container is terminated”, and the kubelet restarts it if the restart policy allows. On nodes with cgroup v2, the kubelet by default makes an OOM kill stop every process in the container together, not only the largest one.
Node out of memory
When the whole node runs out of memory before the kubelet can free any, the kernel’s OOM killer chooses a victim across all containers. The kubelet steers that choice through oom_score_adj, based on the Pod’s quality of service class:
| QoS class | oom_score_adj | Killed |
|---|---|---|
BestEffort (no requests or limits) | 1000 | First |
Burstable | Between 2 and 999, higher for Pods using more memory relative to their request | In between |
Guaranteed (limits equal requests) | -997 | Last |
Node-pressure eviction
The kubelet watches free memory on the node. By default on Linux it starts evicting Pods when available memory drops below 100 MiB. It evicts Pods that use more than their requests first, and with this hard threshold it stops them without a grace period. An evicted Pod is not restarted in place; its phase is Failed and kubectl get pods shows Evicted.
How to tell the causes apart
Read the last termination of the container:
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'reason: OOMKilledpoints to memory.reason: Errorwith code 137 points to a forced kill, usually a probe.Compare the container’s memory limit with its usage before the kill.
kubectl top pod <pod> --containersshows current usage; your metrics system shows the history.Check the node for memory pressure:
kubectl describe node <node>A
MemoryPressurecondition or eviction events point to the node, not the container.If you have node access, the kernel log names the trigger.
Memory cgroup out of memorymeans a container hit its own limit;Out of memorywithout the cgroup prefix means the node itself ran out.
How to fix it
- Set the memory limit from measured peak usage. Watch memory over a full load cycle, then set the limit above the peak with headroom. Setting the request equal to the limit makes the scheduler reserve that memory and gives the Pod the
Guaranteedclass. Kubernetes requests and limits covers the trade-offs. - Make the runtime respect the limit. Garbage-collected runtimes do not always keep their heap below the container limit. Set the heap size explicitly:
-XX:MaxRAMPercentagefor the JVM,--max-old-space-sizefor Node.js,GOMEMLIMITfor Go. - Profile before raising limits. If usage climbs until the kill no matter how high the limit is, the application leaks memory. Take heap profiles or dumps and compare them over time.
- Stop overcommitting nodes. Node OOM kills and evictions happen when the sum of actual usage exceeds the node’s memory, which is likely when requests are far below real usage. Raise requests to realistic values, and reserve memory for system daemons with the kubelet’s
kube-reservedandsystem-reservedsettings. - Tune probes. If the reason is
Error, notOOMKilled, give slow starters astartupProbeand make sure the application exits on SIGTERM within its grace period. - Use recommendations. The Vertical Pod Autoscaler can suggest requests from observed usage; see how Kubernetes autoscaling works.
Memory on Cloudfleet
In Cloudfleet Kubernetes Engine (CFKE), node auto-provisioning sizes new nodes from Pod requests, so realistic memory requests also keep nodes from being packed beyond what their workloads use.