Network blackhole: service unreachable
go-api went dark. Every request through the ingress times out or 5xxes — yet the pods are Running and Ready, logs are quiet, and nothing was deployed. Something is eating the traffic.
What the symptom looks like
go-api went dark. Every request through the ingress times out or 5xxes — yet the pods are Running and Ready, logs are quiet, and nothing was deployed. Something is eating the traffic.
Affects go-api.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose — open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The app says it's fine: pods Running, probes green, logs quiet. So trust the app and suspect the path to it. Walk the request hop by hop: ingress → Service → Endpoints → pod. Where does it die?
Hint 2
`kubectl get endpoints go-api -n go-api` shows healthy endpoints, and `kubectl port-forward` straight to a pod works. So the pod is reachable — but not *through the network path*. What Kubernetes objects can silently drop traffic between two healthy points?
Hint 3
`kubectl get networkpolicy -n go-api`. A policy with `podSelector: {}`, `policyTypes: [Ingress]`, and **no rules** means "select every pod, allow no ingress" — a deny-all. Delete it and the service comes back instantly.
Related incidents
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
- CrashLoop: broken container commandUsers report the latest go-api deploy never finished. New pods start, die instantly, and start again — classic CrashLoopBackOff. The old pods are still serving, so traffic mostly works… for now.
- Noisy neighbor: CPU-hungry tenantLatency on go-api crept up and node CPU is pegged, but go-api itself didn't change. Someone else moved in — a workload with big CPU requests, no limits, and a hot loop.