Bad deploy: image that doesn't exist
A release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
What the symptom looks like
A release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
Affects go-api.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose — open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The deploy "went out" but nothing changed for users. Check the state of the rollout: `kubectl rollout status deploy/go-api -n go-api`. Is it actually finished? Then look at the pods.
Hint 2
A pod stuck in `ImagePullBackOff` or `ErrImagePull` can't even download its container. `kubectl describe pod -n go-api <pod>` — the Events section tells you exactly which image the kubelet tried to pull and why it failed.
Hint 3
Compare the image on the Deployment (`kubectl get deploy go-api -n go-api -o jsonpath='{.spec.template.spec.containers[0].image}'`) with what's actually available. Fix the tag with `kubectl set image` (the previously working image is recorded on the deployment's annotations), or roll back with `kubectl rollout undo`.
Related incidents
- CrashLoop: broken container commandUsers report the latest go-api deploy never finished. New pods start, die instantly, and start again — classic CrashLoopBackOff. The old pods are still serving, so traffic mostly works… for now.
- Network blackhole: service unreachablego-api went dark. Every request through the ingress times out or 5xxes — yet the pods are Running and Ready, logs are quiet, and nothing was deployed. Something is eating the traffic.
- Noisy neighbor: CPU-hungry tenantLatency on go-api crept up and node CPU is pegged, but go-api itself didn't change. Someone else moved in — a workload with big CPU requests, no limits, and a hot loop.