Bad deploy: image that doesn't exist
A release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The deploy "went out" but nothing changed for users. Check the state of the
rollout: kubectl rollout status deploy/go-api -n go-api. Is it actually
finished? Then look at the pods.
Hint 2
A pod stuck in ImagePullBackOff or ErrImagePull can't even download its
container. kubectl describe pod -n go-api <pod> - the Events section
tells you exactly which image the kubelet tried to pull and why it failed.
Hint 3
Compare the image on the Deployment
(kubectl get deploy go-api -n go-api -o jsonpath='{.spec.template.spec.containers[0].image}')
with the one the pod that is still serving is running. The Deployment's own
history remembers the last good release - kubectl rollout history deploy/go-api -n go-api --revision=<n> shows the image each revision used.
Put the working tag back with kubectl set image, or roll the release back
with kubectl rollout undo.
Related incidents
- CrashLoop: a bad config value takes the app downgo-api went down minutes after a config change rolled out. Every request now fails. Every go-api pod starts, dies within a second and starts again (CrashLoopBackOff), and no old pod is left to serve.
- Liveness probe kills a healthy appSince this morning's values update, go-api restarts every 40 seconds or so, and users get bursts of 503s between quiet spells. Every restart looks clean: the logs show the app starting, serving and shutting down on a signal, with no error. The app is not crashing on its own (no exit code 1) and is not being OOMKilled (no exit code 137).
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.