Network blackhole: service unreachable
go-api went dark. Every request through the ingress 5xxes - yet the pods are Running and Ready, the app logs are quiet, and nothing was deployed. The requests are not arriving at all. Something on the wire is eating them.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The page came from the edge, not from the app, and the app says it is fine: pods Running, probes green, logs quiet. Nothing is even reaching it to be logged. Trust the app and suspect the path to it: walk the request hop by hop, ingress → Service → Endpoints → pod, and find where it dies.
Hint 2
kubectl get endpoints go-api -n go-api shows healthy endpoints, and
kubectl port-forward straight to a pod works. So the pod is reachable,
but not through the network path. What Kubernetes objects can silently
drop traffic between two healthy points?
Hint 3
kubectl get networkpolicy -n go-api, then
kubectl describe networkpolicy -n go-api to read what each one
allows. A policy with podSelector: {}, policyTypes: [Ingress], and no
rules means "select every pod, allow no ingress" - a deny-all. Its name will
not tell you it is the problem; its AGE and its empty rule list will. Delete it
and the service comes back instantly.
Related incidents
- DNS egress blocked: a security policy that forgot port 53Since this morning's security hardening, most requests to go-api fail with 502. /health answers, every pod is Ready and nothing restarts, and requests do arrive: unlike a network blackhole, the app answers, with an error from the dependency it calls. That dependency is up and healthy when you call it from anywhere else.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.