SnowOps Labs stands up a production-shaped Kubernetes cluster on your laptop in minutes, breaks it in realistic ways, and grades you on how you fix it. Real Prometheus, real Istio, real failures.
For engineers who passed the CKA and still freeze when the pager goes off.
Build a realistic platform, run a scenario against it, break it on purpose, and measure how you did. Everything else in the project exists to serve these four.
A local k3d cluster plus a real platform stack from swappable providers — ingress, metrics, logs, traces, GitOps, mesh, secrets, autoscaling. Actual Prometheus and actual Istio, at laptop scale.
labctl initEvery fault below is real content from the repo — the same symptom, the same progressive hints, the same scoring. Start the clock and see how far you get before you need help.
What you are told
A release went out referencing an image tag that was never pushed. The rollout is stuck — new pods sit in ImagePullBackOff while the old version keeps serving.
This is a dry run of the loop, not a live cluster. The real thing injects the fault into actual running systems and a machine check decides whether you fixed it.
Declarative scenarios with machine-verifiable checks, reversible faults with progressive hints, and timed challenges with par times — all of it version-controlled alongside the engine.
13 scenarios
Watch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
labctl scenario up autoscaling-under-load1 stageSourceFailure injection with Chaos Mesh — pod kills, network delays, CPU/memory stress, partitions. Correlate chaos experiments with application metrics via Grafana dashboard.
labctl scenario up chaos-engineeringSourceDeploy go-api with deliberately over-provisioned CPU and memory requests, observe the inflated cost in OpenCost, then right-size the requests to a realistic baseline. Checks verify that requests were reduced to the target thresholds while go-api stays healthy under traffic. Teaches the observe-measure-reduce loop that keeps platform costs rational.
labctl scenario up cost-right-sizing1 stageSourceBack up a namespace's resources to a manifest archive, simulate accidental data loss (delete a resource — or the whole namespace), then restore from the backup and verify the round-trip with checks. A stateful data-writer on a PersistentVolume makes the crucial lesson concrete: the manifest backup round-trips Kubernetes OBJECTS, not the DATA on a volume. A dependency-light alternative to Velero that teaches the backup/restore loop every operator must rehearse before they need it.
labctl scenario up backup-restore-drill2 stagesSourceCordon and drain a worker node while go-api serves live traffic. A PodDisruptionBudget keeps a minimum number of replicas available, so the scheduler reschedules pods onto the remaining nodes without dropping requests. The grade is the measured request success rate through the drain.
labctl scenario up node-drain-drill1 stageSourceRoll the cluster's worker nodes to a newer Kubernetes version one at a time while go-api serves live traffic. Each node is drained (respecting the PodDisruptionBudget) before it is replaced, so the application stays available throughout. The grade is the measured request success rate across the upgrade window.
labctl scenario up cluster-upgrade-drill1 stageSourceA producer/consumer flow through Kafka (Strimzi): an 'orders' topic with a continuous producer and a consumer group. Stage 2 ramps producers to build consumer lag — observe it, then scale consumers (manually or with KEDA's Kafka scaler) to drain it.
labctl scenario up event-driven-arch2 stagesSourceArgoCD-based GitOps deployment with Application CRDs for declarative app management.
labctl scenario up gitops-cicdSourceIstio canary release for go-api: a 90/10 weighted split between v1 and v2, mesh-level latency fault injection, and STRICT mTLS on the canary workload. Watch the split and faults in mesh telemetry.
labctl scenario up mesh-traffic-management2 stagesSourceRun a real release pipeline by hand: build versioned go-api images, deploy one to dev, and promote it forward to staging then prod with plain kubectl. Promotion here is a genuine image rollout (new ReplicaSet), not a ConfigMap edit — you build vNEXT, roll it out per environment, and watch each env's /version change as the release moves down the pipeline.
labctl scenario up env-promotion4 stagesSourceFull observability stack with log aggregation, distributed tracing, alerting rules, and SLO dashboards.
labctl scenario up observability-sre3 stagesSourcego-api consumes a Vault-backed secret synced by External Secrets. Stage 1 wires the sync and seeds a baseline value; stage 2 rotates it in Vault. A check verifies the new value propagated to the Kubernetes Secret with no redeploy.
labctl scenario up secrets-management2 stagesSourcePolicy enforcement with Kyverno, TLS certificates via cert-manager, encrypted secrets with Sealed Secrets, namespace network isolation, and security best-practice policies.
labctl scenario up security-complianceSourceActual Prometheus. Actual Istio. Actual Kafka. Failures are injected into real systems, so the dashboards you learn to read are the dashboards you already have at work.
autoscaling
KEDA
chaos
Chaos Mesh
cost
OpenCost
dashboard
Dashboard
data
KafkaPostgreSQL
gitops
ArgoCD
ingress
NginxTraefik
logging
Loki
mesh
IstioLinkerd
monitoring
GrafanaPrometheus
secrets
External secretsVault
security
Network policiesPolicySecretsTls
tracing
Tempo
Categories marked with a slash are exclusive — you pick one ingress, one mesh — which is exactly the evaluation a platform team has to make. Swap the provider, re-run the scenario, compare with evidence instead of opinion.
labctl ui serves an embedded dashboard at localhost:3939 — no extra install, no separate service — and it follows your system light/dark theme.
Production is the only place these skills get exercised, and production is the worst possible classroom.
Learn → scenario → incident
Completes a path and can debug that whole class of failure unaided.
Build → swap provider → compare
Picks Istio over Linkerd with evidence, in a day rather than a month.
Break → measure
The team's MTTR drops measurably across sessions.
Challenge → leaderboard
Objective, reproducible signal on incident-response ability.
Go orchestrates, records and grades — it never encodes the content itself. That is a deliberate design principle, and it is what makes the contribution surface small.
Scenarios, incidents, paths and checks are YAML plus shell scripts. If you can write YAML, you can author for SnowOps Labs — no Go required.
Your first scenarioGood first issues are labelled. The PR bar and the golden rules are written down, so there is no guessing what review will ask for.
Good first issuesDocs live next to the code and this site renders them directly. Fix a line in the repo and it shows up here on the next build.
Contributing guideEarly release, rapid iteration, the odd rough edge. A precise bug report with labctl doctor output is worth a great deal.
Open an issueFree, open source, and running on your laptop in the time it takes to read the quickstart.
labctl doctorOne prerequisite worth knowing before you start: Docker needs at least 4 CPUs and 8 GB of memory. The 2 GB a fresh Docker VM ships with is not enough — labctl doctor checks this and prints the fix for your platform.