CrashLoop: a bad config value takes the app down
go-api went down minutes after a config change rolled out. Every request now fails. Every go-api pod starts, dies within a second and starts again (CrashLoopBackOff), and no old pod is left to serve.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
Users get errors on every request, so start from what should be answering them:
kubectl get pods -n go-api and read the STATUS and RESTARTS
columns. In Grafana, Pod Resources shows the same thing over time:
Container restarts and Containers not running, by reason. Is anything left
serving?
Hint 2
CrashLoopBackOff only says the container keeps exiting, not why. A container
that crashed already wrote its last words: read them with
kubectl logs -n go-api <crashing-pod> --previous, and check
Last State and Exit Code in kubectl describe pod. What does the error
complain about?
Hint 3
The error names a setting the app reads when it starts. Settings like that come
from the pod template, not the pod:
kubectl get deploy go-api -n go-api -o yaml and
read the container's env. Compare it with the containerPort a few lines
above. Fix the Deployment, not the pod: a deleted pod comes back from the same
template and crashes the same way.
Further reading
- Kubernetes - Determine the reason for pod failureLast State, exit codes and termination messages: where a crashed container leaves its evidence.
- Kubernetes - Container restarts and CrashLoopBackOffWhy the wait between restarts doubles up to five minutes.
- Kubernetes - Define environment variables for a containerWhere a container's settings live in the pod template, and how to change them.
Related incidents
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- Liveness probe kills a healthy appSince this morning's values update, go-api restarts every 40 seconds or so, and users get bursts of 503s between quiet spells. Every restart looks clean: the logs show the app starting, serving and shutting down on a signal, with no error. The app is not crashing on its own (no exit code 1) and is not being OOMKilled (no exit code 137).
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.