Missing Secret key: pods that never start
go-api is completely down and every request fails. Two routine jobs ran overnight: the credential rotation, and a node rotation that replaced every pod. No new pod has started since: they are all stuck before their container runs, so there are no logs and no restarts. Nothing was deployed, and unlike a crash loop the app never gets far enough to fail.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
Start from the pods: kubectl get pods -n go-api. Read STATUS:
is any container running at all? A pod whose container never started has no
logs to read, so kubectl logs tells you nothing here.
Hint 2
CreateContainerConfigError means the kubelet could not build the container's
configuration, so it never started it. kubectl describe pod -n go-api <pod> says why in the Events at the bottom. Restarting
the Deployment builds new pods from the same configuration, and they stop in
the same place.
Hint 3
The container reads some of its environment from a Secret, one key at a time
(env[].valueFrom.secretKeyRef in
kubectl get deploy go-api -n go-api -o yaml).
Compare the key it asks for with the keys the Secret has:
kubectl describe secret -n go-api <secret> lists them.
Further reading
- Kubernetes - Distribute credentials securely using SecretsHow a container reads one key of a Secret through env.valueFrom.secretKeyRef.
- Kubernetes - Secrets: using Secrets as environment variablesA reference to a missing key stops the container from starting unless it is marked optional.
- Kubernetes - Debug podsA pod stuck in a Waiting state: describe it and read the reason and the events.
- Kubernetes - Configure a pod to use a ConfigMapThe same failure from a ConfigMap key: CreateContainerConfigError, couldn't find key in ConfigMap.
- Kubernetes - Images: image pull errorsThe other common reason a pod never starts a container: ErrImagePull and ImagePullBackOff.
Related incidents
- Service selector mismatch: empty endpointsThe pager says go-api's Service has no ready endpoints, and every request through the ingress is a 503. But every pod is Running and Ready and nothing was deployed. The Service exists. The pods exist. They just don't know about each other anymore.
- Bad deploy: image that doesn't existA release went out referencing an image tag that was never pushed. The rollout is stuck - new pods sit in ImagePullBackOff while the old version keeps serving.
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.