Pending: replicas no node can hold
We scaled go-api to 4 for a launch, and READY has said 3/4 ever since. Two pods from the change have never started, and the nodes are mostly idle. Nothing is crashing, restarting or being killed, and every request still gets an answer, but there is no headroom for the launch traffic. The pager says the Deployment has fewer replicas available than it asked for.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
Count what is serving and what is not:
kubectl get deploy,rs,pods -n go-api. READY says how many
replicas answer out of how many were asked for. What STATUS are the others in,
and has any of them ever started a container?
Hint 2
A Pending pod has not been placed on a node yet, so it has no logs and no
restarts. Whatever stopped it, the scheduler wrote the reason into the pod's
events: kubectl describe pod -n go-api <pending-pod> and read
the Events at the bottom. Deleting the pod only brings back an identical one.
Hint 3
The scheduler places a pod by its requests, compared with what each node
can allocate (kubectl describe node → Allocatable and Allocated resources),
never by what the pods actually use (kubectl top). Read the CPU request in
the Deployment's pod template
(kubectl get deploy go-api -n go-api -o yaml)
and compare it with a node's allocatable CPU.
Further reading
- Kubernetes - Debug Pods: my pod stays pendingThe first place to look: the scheduler writes why it could not place a pod into the pod's events.
- Kubernetes - Resource management for pods and containersThe scheduler places a pod by its requests against each node's allocatable capacity, never by what the pod uses.
- Kubernetes - Assigning pods to nodesAnother common cause of Pending: a nodeSelector or affinity rule no node satisfies.
- Kubernetes - Taints and tolerationsAnother common cause: every node carries a taint the pod does not tolerate.
- Kubernetes - Persistent volumes: bindingAnother common cause: a PersistentVolumeClaim that never binds keeps its pod Pending.
- Kubernetes - Pod priority and preemptionWhat the 'preemption: … No preemption victims found' half of the scheduler's message means.
Related incidents
- CPU throttling: a limit set to save costgo-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.
- Noisy neighbor: CPU-hungry tenantYou have been paged: the node go-api runs on is out of CPU. Nothing you deployed changed (same image, same replicas, same config) and the other two nodes are idle. Something else moved in, and it is taking the machine.
- OOMKilled: memory limit too lowThe pager says go-api is being killed and restarted over and over. It came up clean and served fine for a minute, then the restart count started climbing and the service began flapping. Nothing was deployed.