Noisy neighbor: CPU-hungry tenant
You have been paged: the node go-api runs on is out of CPU. Nothing you deployed changed (same image, same replicas, same config) and the other two nodes are idle. Something else moved in, and it is taking the machine.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The page named a node, not a workload, so stop looking inside go-api. Start by confirming what the page said, and notice that the other nodes are idle - whatever this is, it is in one place.
kubectl -n go-api get pods -o wide
kubectl top nodesHint 2
The node is full, and go-api is only a small part of it. Something else scheduled there is taking the rest. Rank every pod on the cluster by CPU and look at what is on top, then at what its Deployment asks for.
kubectl top pods -A --sort-by=cpuHint 3
labfault-batch/report-batch requests 500m per replica and sets no CPU
limit. The request bought it a large share of the node's cycles; the missing
limit means nothing stops it taking the rest, so each replica draws about double
what it reserved. go-api keeps serving, its own request guarantees
it a share, but it has lost the headroom to burst to the limit it was given.
Bound it, scale it away, or evict it. Then ask the question that matters: what would have stopped this being possible in the first place?
Related incidents
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.
- OOMKilled: memory limit too lowThe pager says go-api is being killed and restarted over and over. It came up clean and served fine for a minute, then the restart count started climbing and the service began flapping. Nothing was deployed.
- Pending: replicas no node can holdWe scaled go-api to 4 for a launch, and READY has said 3/4 ever since. Two pods from the change have never started, and the nodes are mostly idle. Nothing is crashing, restarting or being killed, and every request still gets an answer, but there is no headroom for the launch traffic. The pager says the Deployment has fewer replicas available than it asked for.