Service selector mismatch: empty endpoints
The pager says go-api's Service has no ready endpoints, and every request through the ingress is a 503. But every pod is Running and Ready and nothing was deployed. The Service exists. The pods exist. They just don't know about each other anymore.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The page named the Service, not a pod, and 503 from an ingress usually means it had nowhere to send the request. Pods look fine, so check the layer between ingress and pods. Is the Service actually fronting anything?
Hint 2
kubectl get endpoints go-api -n go-api: <none>. A Service with no
endpoints matches no pods. Endpoints come from the Service's
selector matching pod labels. Compare them.
Hint 3
kubectl get svc go-api -n go-api -o jsonpath='{.spec.selector}' vs
kubectl get pods -n go-api --show-labels. They disagree on
app.kubernetes.io/name: the Service is looking for a value the pods do not
carry, and a selector is an AND, so one wrong key matches nothing.
Fix it on the Service, not the pods. Relabelling a pod to match makes the 503s stop and leaves the Deployment still stamping the other label on every pod it creates, so the next rollout breaks it again.
Related incidents
- Missing Secret key: pods that never startgo-api is completely down and every request fails. Two routine jobs ran overnight: the credential rotation, and a node rotation that replaced every pod. No new pod has started since: they are all stuck before their container runs, so there are no logs and no restarts. Nothing was deployed, and unlike a crash loop the app never gets far enough to fail.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.