Service selector mismatch: empty endpoints
go-api returns 503s through the ingress, but every pod is Running and Ready and nothing was deployed. The Service exists. The pods exist. They just don't know about each other anymore.
What the symptom looks like
go-api returns 503s through the ingress, but every pod is Running and Ready and nothing was deployed. The Service exists. The pods exist. They just don't know about each other anymore.
Affects go-api.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose — open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
503 from the ingress usually means the ingress had nowhere to send the request. Pods look fine, so check the layer between ingress and pods: the Service. Is it actually fronting anything?
Hint 2
`kubectl get endpoints go-api -n go-api` — `<none>`. A Service with no endpoints matches no pods. Endpoints come from the Service's **selector** matching pod **labels**. Compare them.
Hint 3
`kubectl get svc go-api -n go-api -o jsonpath='{.spec.selector}'` vs `kubectl get pods -n go-api --show-labels`. The selector says `app.kubernetes.io/name=labfault-nobody`; the pods say `app.kubernetes.io/name=go-api`. Patch the selector back and watch the endpoints repopulate.
Related incidents
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
- CrashLoop: broken container commandUsers report the latest go-api deploy never finished. New pods start, die instantly, and start again — classic CrashLoopBackOff. The old pods are still serving, so traffic mostly works… for now.
- Network blackhole: service unreachablego-api went dark. Every request through the ingress times out or 5xxes — yet the pods are Running and Ready, logs are quiet, and nothing was deployed. Something is eating the traffic.