All of the catalog
Scenario
Observability & SRE
labctl stands up the three signals (Loki for logs, Tempo behind an Alloy collector for traces, Prometheus rules and SLO dashboards for metrics) and then leaves the wiring to you. Export go-api's spans through the collector, drive real load with k6, follow one request from a log line to its trace, and take an SRE alert from inactive to firing by breaking readiness on purpose.
ObservabilityVerifiedk3dkind
Definition on GitHubWhat you'll do
- Wire go-api to the collector and prove spans reach Tempo - the app exports OTLP to Alloy and never learns which backend is behind it
- Drive real load with k6 and confirm the app's own request metrics move with it
- Correlate the three signals: find a log line's trace_id in Loki and open that trace in Tempo
- Break readiness under load and drive an SRE alert from inactive to firing
Stages
- 1log-aggregation
Ship and store application logs
lokipromtail - 2tracing
Trace collector (Alloy) and trace backend (Tempo)
tempoalloyreset-tracing-wiring - 3alerting-and-slo
Alert rules and SLO dashboards
alerting-rulesslo-dashboards
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressmonitoring/metricsmonitoring/grafanago-api
Related scenarios
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
- Chaos EngineeringRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
- Cost & Capacity: Right-Sizing Over-Provisioned Resourcesgo-api arrives reserving 2000m CPU and 1Gi of memory - forty times what it uses. Drive real traffic, read the gap between reserved and used on the dashboard and in OpenCost, then right-size the requests. Grading has two sides: a request over the waste ceiling fails, and so does one under the peak the workload actually reached, because under-provisioning is how workloads get throttled and OOM-killed.