CPU throttling: a limit set to save cost
go-api has got slow since the cost review's resource changes went out, and the latency SLO is burning: p99 is in seconds while the median still looks fine. Nothing restarts, nothing is OOMKilled, every pod is Running and Ready, and the error rate is flat. The node it runs on is nearly idle, so this is not a crowded machine.
Which app it breaks
go-api by default. Pass --app to break another app in your lab instead.
The hint ladder
Each hint narrows the search a little further. They are collapsed on purpose: open one only when you are properly stuck, because working out the next command yourself is the whole exercise.
Hint 1
Nothing is broken, it is slow, so look at time rather than errors. In Grafana,
Application Request Metrics → Latency percentiles: compare p99 with p50.
A healthy app that is slow for 1% of requests is waiting for something. Then
rule out the obvious: kubectl get pods -n go-api (restarts?)
and kubectl top nodes (a busy node?).
Hint 2
kubectl top pod -n go-api shows how much CPU each pod uses.
Compare it with the container's CPU limit
(kubectl describe pod -n go-api <pod> → Limits). A pod
using exactly its limit is not idle and not overloaded: it is capped, and the
number you see is the ceiling, not what it wants.
Hint 3
A CPU limit is enforced as a quota per 100ms period; a container that spends
its quota early waits for the next period, and the requests it was serving
wait with it. Pod Resources → CPU throttled shows the share of periods
that ran out. The limit in the Deployment
(kubectl get deploy go-api -n go-api -o yaml) is
below what the app needs under this load. You cannot measure that while it is
capped, so lift the limit first, then size it.
Further reading
- Kubernetes - Assign CPU resources to containers and podsWhat a CPU limit does: the container is not killed, it is held back.
- Kubernetes - Resource management: how limits are appliedA CPU limit becomes a CFS quota per 100ms period; a container that uses it up waits for the next period.
- Kubernetes - Resource metrics pipelineWhere kubectl top's numbers come from, and why a capped container reads exactly its limit.
- Kubernetes - Node-pressure evictionOther causes of slow pods on a busy node: contention for the node's CPU, memory pressure.
- Kubernetes - Horizontal Pod AutoscalingAnother common cause of latency under load: too few replicas for the traffic, which is a different fix.
Related incidents
- Noisy neighbor: CPU-hungry tenantYou have been paged: the node go-api runs on is out of CPU. Nothing you deployed changed (same image, same replicas, same config) and the other two nodes are idle. Something else moved in, and it is taking the machine.
- OOMKilled: memory limit too lowThe pager says go-api is being killed and restarted over and over. It came up clean and served fine for a minute, then the restart count started climbing and the service began flapping. Nothing was deployed.
- Pending: replicas no node can holdWe scaled go-api to 4 for a launch, and READY has said 3/4 ever since. Two pods from the change have never started, and the nodes are mostly idle. Nothing is crashing, restarting or being killed, and every request still gets an answer, but there is no headroom for the launch traffic. The pager says the Deployment has fewer replicas available than it asked for.