DNS egress blocked: a security policy that forgot port 53
Since this morning's security hardening, most requests to go-api fail with 502. /health answers, every pod is Ready and nothing restarts, and requests do arrive: unlike a network blackhole, the app answers, with an error from the dependency it calls. That dependency is up and healthy when you call it from anywhere else.
Which app it breaks
go-api, echo-server by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
The app is answering, so ask it what is wrong: curl -s http://go-api.snowops.localhost/ prints the error it returns, and
kubectl logs -n go-api deploy/go-api shows the same line for every failed
request. Read the whole error: which step of the call to the dependency
failed?
Hint 2
The error is a failed lookup: the app never got as far as connecting,
because the dependency's name did not resolve. Try the same lookup from a
throwaway pod in the same namespace, then from one in another namespace:
kubectl -n go-api run dnstest --rm -it --restart=Never --image=busybox:1.36 -- sh -c 'sleep 5; nslookup echo-server.echo-server.svc.cluster.local'. Does it
work in one and not the other? (Keep the sleep: the network policy controller
needs a moment to start enforcing on a new pod, and a lookup in its first
second can get through.)
Hint 3
Something applies to every pod in the go-api namespace and not elsewhere:
kubectl get networkpolicy -n go-api. A policy that lists Egress in its
policyTypes allows only the traffic its egress rules name, and every name
lookup is traffic to the cluster DNS Service on port 53. Read the rules with
kubectl get networkpolicy -n go-api -o yaml.
Further reading
- Kubernetes - Network policiesA policy that selects a pod for Egress allows only what its rules list, and that includes DNS.
- Kubernetes - Declare network policy: default deny egressThe baseline this fault starts from, and why it needs a DNS rule beside it.
- Kubernetes - Debugging DNS resolutionTesting name resolution from a throwaway pod with nslookup, and the other causes of failed lookups.
- Kubernetes - DNS for Services and PodsHow a Service name resolves, and why every lookup goes to the cluster DNS Service on port 53.
- Kubernetes - Debug ServicesOther causes of a dependency that cannot be reached: no endpoints, the wrong port, a CoreDNS outage.
Related incidents
- Network blackhole: service unreachablego-api went dark. Every request through the ingress 5xxes - yet the pods are Running and Ready, the app logs are quiet, and nothing was deployed. The requests are not arriving at all. Something on the wire is eating them.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.