All of the catalog
Scenario
Day-2 Drill: Node Drain Under Load
Take a worker node out of service while go-api serves live traffic. labctl gives the workload enough replicas to survive; you write the PodDisruptionBudget, cordon and drain the node with kubectl, watch the pods reschedule, and put the node back. The grade is the measured request success rate across the window.
OperationsVerifiedk3dkind
Definition on GitHubWhat you'll do
- Write and apply a PodDisruptionBudget that keeps go-api serving while a node is evicted
- Cordon and drain a worker node with kubectl while traffic runs, and watch pods reschedule
- Prove the request success rate held across the drain, from Prometheus
- Return the node to service and leave the cluster at full capacity
Stages
- 1baseline
Scale go-api so a PodDisruptionBudget has something to protect. The budget itself is yours to write.
workload-ha-replicasdrain-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressmonitoring/metricsgo-api
Related scenarios
- Day-2 Drill: Namespace Backup & RestoreRehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.
- Day-2 Drill: Rolling Cluster Upgrade Under LoadRoll the cluster's worker nodes onto a new Kubernetes version one at a time while go-api serves live traffic. You write the PodDisruptionBudget, you cordon and drain each node, and you watch the PDB block the eviction that would have taken the app down. The grade is the measured request success rate across the upgrade window and a cluster left with no node behind.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.