Noisy neighbor: CPU-hungry tenant
Latency on go-api crept up and node CPU is pegged, but go-api itself didn't change. Someone else moved in — a workload with big CPU requests, no limits, and a hot loop.
What the symptom looks like
Latency on go-api crept up and node CPU is pegged, but go-api itself didn't change. Someone else moved in — a workload with big CPU requests, no limits, and a hot loop.
Affects go-api.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose — open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
go-api didn't change, but the *node* it runs on did. Look at cluster-level resource usage: `kubectl top nodes` (or the node CPU panel in Grafana). Who is using all that CPU?
Hint 2
`kubectl top pods -A --sort-by=cpu` ranks every pod on the cluster by CPU. The top entries aren't yours. Which namespace are they in, and what do their resource requests/limits look like?
Hint 3
A deployment with `requests.cpu: 500m` per replica and **no CPU limit** in the `labfault-noisy-neighbor` namespace is burning everything it can grab. Evict it (`kubectl delete namespace labfault-noisy-neighbor`) — and think about what would have prevented this: limits, quotas, or LimitRanges.
Related incidents
- OOMKilled: memory limit too lowecho-server pods keep restarting and the service is flapping. Restart counts climb; the kernel is killing the container the moment it allocates memory.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
- CrashLoop: broken container commandUsers report the latest go-api deploy never finished. New pods start, die instantly, and start again — classic CrashLoopBackOff. The old pods are still serving, so traffic mostly works… for now.