Liveness probe kills a healthy app
Since this morning's values update, go-api restarts every 40 seconds or so, and users get bursts of 503s between quiet spells. Every restart looks clean: the logs show the app starting, serving and shutting down on a signal, with no error. The app is not crashing on its own (no exit code 1) and is not being OOMKilled (no exit code 137).
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The restarts are the place to start: kubectl get pods -n go-api
shows RESTARTS climbing. Before you read any app logs, find out how the last
container ended: kubectl describe pod -n go-api <pod> and read
Last State, its Reason and Exit Code. Did the app exit on its own?
Hint 2
Exit code 0 and Reason: Completed mean the app shut down cleanly because it
was told to: the logs end with it receiving a signal. When the kubelet restarts
a container, it records why in the pod's events. Read the Events at the bottom
of kubectl describe pod, or
kubectl get events -n go-api --sort-by=.lastTimestamp.
Hint 3
The kubelet restarts a container when its liveness probe fails
failureThreshold times in a row. Compare the probe in the Deployment
(kubectl get deploy go-api -n go-api -o yaml, under
livenessProbe) with the paths the app really serves. Its own logs show which
path it answers 404 to, every ten seconds.
Further reading
- Kubernetes - Configure liveness, readiness and startup probesHow the kubelet runs each probe, and what it does when a liveness probe fails failureThreshold times in a row.
- Kubernetes - Pod lifecycle: container probesWhich HTTP status codes count as success (200 to 399), and the difference between liveness, readiness and startup probes.
- Kubernetes - Determine the reason for pod failureReading Last State and the exit code to tell a crash, an OOM kill and a kubelet restart apart.
- Kubernetes - Debug running podsOther causes of a liveness kill: a probe timeout under load, a slow start with no startupProbe, a probe that checks a dependency.
Related incidents
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- CrashLoop: a bad config value takes the app downgo-api went down minutes after a config change rolled out. Every request now fails. Every go-api pod starts, dies within a second and starts again (CrashLoopBackOff), and no old pod is left to serve.
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.