All of the catalog
Scenario
Chaos Engineering
Run the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
ReliabilityVerifiedk3dkind
Definition on GitHubWhat you'll do
- Establish a steady state: put continuous load on go-api with 'labctl traffic' and read baseline RPS, error rate and p99 off the chaos dashboard
- Run a pod-kill experiment against the single-replica go-api and measure the blast radius - the requests that fail before Kubernetes restarts the pod
- Read the control that is missing: 'kubectl -n go-api get pdb' reports disruptionsAllowed 0 / InsufficientPods, so the PDB blocks every drain instead of protecting anything
- Harden the workload with 'kubectl -n go-api scale deploy/go-api --replicas=3' until the PDB reports disruptionsAllowed >= 1
- Re-run the same pod-kill experiment against the hardened service and prove the outage is gone while the restart still shows on the dashboard
- Explore the other failure modes (CPU stress, memory stress, network delay and partition) and find the signal that reveals each one, including the two the dashboard cannot show you
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressmonitoring/metricsmonitoring/grafanachaos/chaos-meshgo-api
Related scenarios
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
- Cost & Capacity: Right-Sizing Over-Provisioned Resourcesgo-api arrives reserving 2000m CPU and 1Gi of memory - forty times what it uses. Drive real traffic, read the gap between reserved and used on the dashboard and in OpenCost, then right-size the requests. Grading has two sides: a request over the waste ceiling fails, and so does one under the peak the workload actually reached, because under-provisioning is how workloads get throttled and OOM-killed.
- Day-2 Drill: Namespace Backup & RestoreRehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.