Stray pod behind the Service: a fraction of requests fail
The pager says go-api is burning its availability SLO: a steady fraction of requests fails, not all of them. Every pod is Running and Ready, nothing was deployed, and the app's own error-rate panel is flat. Clients see the errors; the app's metrics do not.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
Start from the dashboard. On Application Request Metrics, the load generator's failed-request rate is a steady fraction while the app's own 5xx rate is flat. Some requests fail somewhere the app's metrics cannot see. What else is the Service sending traffic to? Compare its backends with the Deployment's replicas.
Hint 2
kubectl get endpoints go-api -n go-api should have one
address for each Deployment replica. If it has one more, list the pods with
their labels and ages. One may not look like a pod the Deployment created.
Hint 3
The extra pod has the Service's labels but no pod-template-hash. Describe it
and look for Controlled By, then read its logs. Its /ready answers 200 even
though its upstream fails, so readiness keeps it in the Service while /
returns 5xx. What stops the Service sending it traffic, without touching the
Deployment?
Further reading
- Kubernetes - ServiceA Service sends traffic to every ready pod whose labels match its selector, whichever controller created the pod.
- Kubernetes - Owners and dependentsOwner references show which controller owns a Kubernetes object.
- Kubernetes - Debug ServicesThe endpoints and pods are the first places to look when a Service behaves unexpectedly.
- Kubernetes - Debug podsReading a pod's events and logs. Other causes of a steady fraction of failures: one bad replica of a rollout in progress, or one node with a broken network path.
Related incidents
- DNS egress blocked: a security policy that forgot port 53Since this morning's security hardening, most requests to go-api fail with 502. /health answers, every pod is Ready and nothing restarts, and requests do arrive: unlike a network blackhole, the app answers, with an error from the dependency it calls. That dependency is up and healthy when you call it from anywhere else.
- Network blackhole: service unreachablego-api went dark. Every request through the ingress 5xxes - yet the pods are Running and Ready, the app logs are quiet, and nothing was deployed. The requests are not arriving at all. Something on the wire is eating them.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.