SnowOps Labs stands up a production-shaped Kubernetes cluster on your laptop in minutes, breaks it in realistic ways, and grades you on how you fix it. Real Prometheus, real Istio, real failures.
For engineers who passed the CKA and still freeze when the pager goes off.
Build a realistic platform, run a scenario against it, break it on purpose, and measure how you did. Everything else in the project exists to serve these four.
A local k3d cluster plus a real platform stack from swappable providers — ingress, metrics, logs, traces, GitOps, mesh, secrets, autoscaling. Actual Prometheus and actual Istio, at laptop scale.
labctl initEvery fault below is real content from the repo — the same symptom, the same progressive hints, the same scoring. Start the clock and see how far you get before you need help.
What you are told
A release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
This is a dry run of the loop, not a live cluster. The real thing injects the fault into actual running systems and a machine check decides whether you fixed it.
Declarative scenarios with machine-verifiable checks, reversible faults with progressive hints, and timed challenges with par times — all of it version-controlled alongside the engine.
13 scenarios
Watch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
labctl scenario up autoscaling-under-load1 stageSourceRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure — pod kill, network delay, CPU and memory stress — and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
labctl scenario up chaos-engineeringSourceDeploy go-api with deliberately over-provisioned CPU and memory requests, observe the inflated cost in OpenCost, then right-size the requests to a realistic baseline. Checks verify that requests were reduced to the target thresholds while go-api stays healthy under traffic. Teaches the observe-measure-reduce loop that keeps platform costs rational.
labctl scenario up cost-right-sizing1 stageSourceBack up a namespace's resources to a manifest archive, simulate accidental data loss (delete a resource — or the whole namespace), then restore from the backup and verify the round-trip with checks. A stateful data-writer on a PersistentVolume makes the crucial lesson concrete: the manifest backup round-trips Kubernetes OBJECTS, not the DATA on a volume. A dependency-light alternative to Velero that teaches the backup/restore loop every operator must rehearse before they need it.
labctl scenario up backup-restore-drill2 stagesSourceCordon and drain a worker node while go-api serves live traffic. A PodDisruptionBudget keeps a minimum number of replicas available, so the scheduler reschedules pods onto the remaining nodes without dropping requests. The grade is the measured request success rate through the drain.
labctl scenario up node-drain-drill1 stageSourceRoll the cluster's worker nodes onto a new Kubernetes version one at a time while go-api serves live traffic. You write the PodDisruptionBudget, you cordon and drain each node, and you watch the PDB block the eviction that would have taken the app down. The grade is the measured request success rate across the upgrade window and a cluster left with no node behind.
labctl scenario up cluster-upgrade-drill1 stageSourceA producer/consumer flow through Kafka (Strimzi): an 'orders' topic with a continuous producer and a consumer group. Stage 2 ramps producers to build consumer lag — observe it, then scale consumers (manually or with KEDA's Kafka scaler) to drain it.
labctl scenario up event-driven-arch2 stagesSourceRun a real GitOps loop against a Git repository you can actually push to. The lab serves its own repo in-cluster; ArgoCD reconciles it. You clone it, change the declared image and replica count, push, and watch the cluster follow — then prove the two halves of the contract that make GitOps different from kubectl: self-heal reverts a change you make by hand, and prune deletes a resource you remove from Git.
labctl scenario up gitops-cicd4 stagesSourceIstio canary release for go-api: a 90/10 weighted split between v1 and v2, mesh-level latency fault injection, and STRICT mTLS on the canary workload. Watch the split and faults in mesh telemetry.
labctl scenario up mesh-traffic-management2 stagesSourceRun a real release pipeline by hand: build versioned go-api images, deploy one to dev, and promote it forward to staging then prod with plain kubectl. Promotion here is a genuine image rollout (new ReplicaSet), not a ConfigMap edit — you build vNEXT, roll it out per environment, and watch each env's /version change as the release moves down the pipeline.
labctl scenario up env-promotion4 stagesSourceFull observability stack with log aggregation, distributed tracing, alerting rules, and SLO dashboards.
labctl scenario up observability-sre3 stagesSourceA workload mounts a Vault-backed secret that External Secrets keeps in sync. You rotate the value in Vault; it must reach the running pod through ESO and the kubelet with no restart and no redeploy. The grade is what the running container reads from disk.
labctl scenario up secrets-management1 stageSourceTake a cluster from 'policies exist' to 'policies bite'. Kyverno ships four Pod Security and best-practice ClusterPolicies in Audit mode, so nothing is blocked and every violation lands in a policy report. Your job is the part a real team owns: read the report, remediate the workload at the source, promote the policies to Enforce, prove the admission webhook now rejects a non-compliant Pod, and issue a real TLS certificate from the lab CA with cert-manager — on a namespace that is default-deny in both directions.
labctl scenario up security-compliance4 stagesSourceActual Prometheus. Actual Istio. Actual Kafka. Failures are injected into real systems, so the dashboards you learn to read are the dashboards you already have at work.
_lib
autoscaling
KEDA
chaos
Chaos Mesh
cost
OpenCost
dashboard
Dashboard
data
KafkaPostgreSQL
gitops
ArgoCD
ingress
NginxTraefik
logging
Loki
mesh
IstioLinkerd
monitoring
GrafanaPrometheus
secrets
External secretsVault
security
Network policiesPolicySecretsTls
tracing
Tempo
Categories marked with a slash are exclusive — you pick one ingress, one mesh — which is exactly the evaluation a platform team has to make. Swap the provider, re-run the scenario, compare with evidence instead of opinion.
labctl ui serves an embedded dashboard at localhost:3939 — no extra install, no separate service — and it follows your system light/dark theme.
Production is the only place these skills get exercised, and production is the worst possible classroom.
Learn → scenario → incident
Completes a path and can debug that whole class of failure unaided.
Build → swap provider → compare
Picks Istio over Linkerd with evidence, in a day rather than a month.
Break → measure
The team's MTTR drops measurably across sessions.
Challenge → leaderboard
Objective, reproducible signal on incident-response ability.
Go orchestrates, records and grades — it never encodes the content itself. That is a deliberate design principle, and it is what makes the contribution surface small.
Scenarios, incidents, paths and checks are YAML plus shell scripts. If you can write YAML, you can author for SnowOps Labs — no Go required.
Your first scenarioGood first issues are labelled. The PR bar and the golden rules are written down, so there is no guessing what review will ask for.
Good first issuesDocs live next to the code and this site renders them directly. Fix a line in the repo and it shows up here on the next build.
Contributing guideEarly release, rapid iteration, the odd rough edge. A precise bug report with labctl doctor output is worth a great deal.
Open an issueFree, open source, and running on your laptop in the time it takes to read the quickstart.
labctl doctorOne prerequisite worth knowing before you start: Docker needs at least 4 CPUs and 8 GB of memory. The 2 GB a fresh Docker VM ships with is not enough — labctl doctor checks this and prints the fix for your platform.
One real Kubernetes failure a week — the symptom, the commands that found it, and the fix. Written from actual lab runs, not from memory.
You'll get the Kubernetes Incident Response Field Guide, plus occasional emails about new scenarios, posts and paid offerings such as courses and workshops. Unsubscribe any time.