A pod restarts every few minutes and kubectl get pods shows OOMKilled in the status column, or just a climbing restart count. The quick fix is to double the memory limit and move on. Sometimes that's right. Often it only buys a few days, because the limit was never the problem.
Kubernetes has three different ways of killing a workload for memory, and they leave different traces. This guide walks through telling them apart, finding out what actually used the memory, and choosing between a bigger limit, a runtime setting and a code fix. Everything here applies to current releases (Kubernetes 1.37 at the time of writing) on Linux nodes with cgroup v2.
Three ways memory kills a pod
The first is a container OOM kill. Each container with a memory limit gets its own cgroup, and the kernel enforces that limit. When the container's usage hits the limit and the kernel can't reclaim enough page cache, the kernel's OOM killer kills it. The pod stays on the node and the kubelet restarts the container according to the pod's restartPolicy.
The second is a node OOM kill. Here no single container hit its limit, but the node as a whole ran out of memory before the kubelet could react. The kernel picks a victim across the whole machine, guided by an oom_score_adj value the kubelet assigns to every container based on its QoS class.
The third is a kubelet eviction. The kubelet watches node memory and, when memory.available drops below a threshold (100Mi by default on Linux), it evicts whole pods to protect the node. An evicted pod is not restarted. Its phase becomes Failed and its controller creates a replacement, possibly on another node.
The fix for each is different, so identify which one you have before changing anything.
Step 1: read the last termination state
The container's previous run is recorded in the pod status under lastState. kubectl describe shows it in readable form.
kubectl describe pod api-7d9c6b8f4-x2klp -n shop
State: Running
Started: Thu, 12 Nov 2026 09:41:07 +0100
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 12 Nov 2026 09:33:52 +0100
Finished: Thu, 12 Nov 2026 09:41:05 +0100
Ready: True
Restart Count: 6
Limits:
memory: 512Mi
Requests:
cpu: 250m
memory: 256Mi
Exit code 137 means the process died from signal 9 (SIGKILL), since shell convention reports a signal death as 128 plus the signal number. SIGKILL alone doesn't prove an OOM kill, because the kubelet also sends it when a container ignores SIGTERM past its grace period. The OOMKilled reason is the part that matters. The container runtime sets it when the kernel recorded an OOM kill in the container's cgroup.
When a Deployment has many replicas, find all the affected containers at once with jq.
kubectl get pods -A -o json | jq -r '
.items[]
| .metadata.namespace as $ns
| .metadata.name as $pod
| .status.containerStatuses[]?
| select(.lastState.terminated.reason == "OOMKilled")
| "\($ns)/\($pod) \(.name) restarts=\(.restartCount)"'
If the reason is OOMKilled and the container sits on a limit, as above, you're dealing with a container OOM kill. Skip to step 3.
Step 2: rule out evictions and node OOM kills
Evicted pods look different. They don't restart; they sit in the Failed phase with the reason Evicted, and the message names the resource the node was short of.
kubectl get pods -A --field-selector=status.phase=Failed
kubectl get pod api-7d9c6b8f4-q8wzn -n shop -o jsonpath='{.status.reason}{"\n"}{.status.message}{"\n"}'
Evicted
The node was low on resource: memory. Threshold quantity: 100Mi, available: 84312Ki. Container api was using 1123044Ki, request is 256Mi, has larger consumption of memory.
That last sentence is useful. The kubelet ranks eviction candidates first by whether their usage exceeds their requests, then by priority, then by how far over their requests they are. A container using 1 GiB against a 256 MiB request is a natural first victim. Note also that node-pressure eviction ignores PodDisruptionBudgets, and with hard thresholds it uses a zero-second grace period.
A node OOM kill leaves an event on the node rather than on the pod. The kubelet watches the kernel log and records a SystemOOM warning when the OOM killer fires outside any container's own limit.
kubectl get events -A --field-selector=reason=SystemOOM
kubectl describe node worker-3 | grep -A3 -i oom
Warning SystemOOM 4m kubelet System OOM encountered, victim process: java, pid: 48211
Both evictions and node OOM kills point at the same root cause. Pods on that node use far more memory than they requested, so the scheduler packed the node based on numbers that don't reflect reality. Raising limits won't help here. Raising requests to match real usage will.
Step 3: measure what the container really uses
Before you change a limit, find out how close normal usage runs to it. kubectl top reads from metrics-server, which reports the working set, an estimate of the memory that can't be freed under pressure.
kubectl top pod -n shop -l app=api --containers
POD NAME CPU(cores) MEMORY(bytes)
api-7d9c6b8f4-x2klp api 212m 488Mi
api-7d9c6b8f4-m4tqz api 187m 301Mi
api-7d9c6b8f4-v7hcd api 204m 455Mi
A one-off reading tells you little. What you want is the shape over time, and that comes from your Prometheus or whatever scrapes cAdvisor, using container_memory_working_set_bytes. A steady climb that resets at each restart is a leak. A flat line with sharp spikes is a burst, usually a large request, a batch job or a cache warming up. A flat line just below the limit is a container that was simply sized too small.
For a closer look, read the cgroup files from inside the container. On cgroup v2 they sit directly under /sys/fs/cgroup.
kubectl exec -n shop api-7d9c6b8f4-x2klp -c api -- cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current
kubectl exec -n shop api-7d9c6b8f4-x2klp -c api -- cat /sys/fs/cgroup/memory.events
536870912
511705088
low 0
high 0
max 1843
oom 6
oom_kill 6
oom_group_kill 6
memory.max is the 512 MiB limit in bytes. In memory.events, max counts how often usage hit the limit and the kernel had to reclaim, oom counts how often reclaim failed, and oom_kill counts processes killed. A large max with few oom events means the container lives right at its limit and survives on page cache reclaim, which usually also shows up as poor latency.
Remember that memory isn't only your heap. Files written to an emptyDir with medium: Memory live in tmpfs and count against the memory limit of the container that wrote them. Logs or temporary uploads spooled to a RAM-backed volume are a classic surprise.
Step 4: know how the kernel picks what to kill
On cgroup v2 nodes the kubelet sets memory.oom.group on each container's cgroup. When the OOM killer fires in that cgroup, it kills every process in the container together instead of picking one. This is a deliberate choice. A web server with worker processes would otherwise lose one worker and limp on in a broken state, with no restart and no OOMKilled status to tell you.
If you run something that is designed to survive losing a child process, such as a supervisor that respawns workers, you can restore the old per-process behavior with the singleProcessOOMKill field in the kubelet configuration. It applies to the whole node, so most clusters should leave it at its default of false.
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
singleProcessOOMKill: true
For node OOM kills, the victim depends on QoS. Containers in Guaranteed pods get an oom_score_adj of -997, BestEffort containers get 1000, and Burstable containers get a value between 2 and 999 that drops as their memory request grows relative to node capacity. So a Burstable container with a tiny request and a large footprint is the first to go when the node runs dry.
Step 5: fix the runtime before the limit
Many OOM kills come from a runtime that doesn't know it has a limit, or sizes itself as if it had the whole limit for the heap. Fix that first, because it changes what any limit means.
Modern JDKs read the container's cgroup limit, but the default maximum heap is only 25% of it. Teams then set -Xmx close to the limit and forget that metaspace, thread stacks, direct buffers and the JIT live outside the heap. Setting the heap as a percentage keeps it proportional when you change the limit.
containers:
- name: api
image: registry.example.com/shop/api:2.4.0
env:
- name: JAVA_TOOL_OPTIONS
value: "-XX:MaxRAMPercentage=75.0"
resources:
requests:
memory: 768Mi
limits:
memory: 768Mi
Go programs can set a soft limit with GOMEMLIMIT, which makes the garbage collector work harder as usage approaches it. Set it around 90% of the container limit so the collector reacts before the kernel does. Node.js has its own heap cap through --max-old-space-size, in megabytes.
If memory climbs steadily across hours no matter the setting, you have a leak, and no limit will fix it. Take a heap dump or profile before the next kill and fix the code.
Step 6: set requests and limits from real data
Once the runtime is configured, size the container from what you measured. A reasonable starting point is a limit 20 to 30% above the peak working set you saw over a week of normal traffic, and a request at or close to typical usage.
Setting the memory request equal to the limit is worth considering for anything important. Memory is not compressible like CPU, so a container that bursts above its request is borrowing memory the scheduler promised to someone else. Equal requests and limits (on CPU as well) make the pod Guaranteed, which gives it the lowest OOM priority and puts it last in line for eviction. The cost is that you pay for the peak all the time.
On Kubernetes 1.33 and later you can change a running pod's resources in place through the resize subresource, which needs a kubectl client of at least 1.32. This is handy to confirm a new limit stops the kills before you change the Deployment.
kubectl patch pod api-7d9c6b8f4-x2klp -n shop --subresource resize --patch \
'{"spec":{"containers":[{"name":"api","resources":{"requests":{"memory":"768Mi"},"limits":{"memory":"768Mi"}}}]}}'
The kubelet makes only a best-effort check when you lower a memory limit, so don't shrink a container that is already near its new value. And a pod-level resize is temporary by nature, since the next rollout recreates the pod from the Deployment template. Once the numbers hold, put them in the template.
kubectl set resources deployment/api -n shop -c api --requests=memory=768Mi --limits=memory=768Mi
Finally, keep an alert on OOMKilled restarts rather than discovering them during an incident. A container that is OOM-killed once a week is telling you its limit sits right at the edge of normal traffic, and the next traffic spike will make it fail far more often.