OOMKilled: memory limit too low
The pager says go-api is being killed and restarted over and over. It came up clean and served fine for a minute, then the restart count started climbing and the service began flapping. Nothing was deployed.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The pods are restarting on a loop. kubectl get pods -n go-api shows
climbing RESTARTS, but why are they dying? kubectl describe pod and
read the container's Last State carefully.
Hint 2
Last State: Terminated, Reason: OOMKilled, Exit Code: 137. The kernel is
killing the container for exceeding its memory limit. So what is the
limit? Check the deployment's resources block.
Hint 3
The limit was cut to just above what the container uses at rest: enough to
start on and idle on, which is why nothing looked wrong for the first minute,
but not enough to serve requests. Read the number off the deployment itself rather
than assuming one, then compare it with what the app actually uses under load
(Grafana's container memory panels, or kubectl top pod -n go-api
while the load is running) and raise the limit above that peak.
Further reading
- Kubernetes - Assign memory resources to containersrequests vs limits, and what the kernel does when a container exceeds its memory limit.
- Kubernetes - Debug an OOMKilled container / exit code 137
- Prometheus - kube-state-metrics container restart & OOM signals
Related incidents
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.
- Noisy neighbor: CPU-hungry tenantYou have been paged: the node go-api runs on is out of CPU. Nothing you deployed changed (same image, same replicas, same config) and the other two nodes are idle. Something else moved in, and it is taking the machine.
- Pending: replicas no node can holdWe scaled go-api to 4 for a launch, and READY has said 3/4 ever since. Two pods from the change have never started, and the nodes are mostly idle. Nothing is crashing, restarting or being killed, and every request still gets an answer, but there is no headroom for the launch traffic. The pager says the Deployment has fewer replicas available than it asked for.