All of the catalog
Scenario
Event-Driven Architecture
An 'orders' topic on Kafka (Strimzi) with a producer/consumer flow already running, and producers ramped to five times what the consumer group can process. Orders are piling up. Diagnose the backlog from Kafka's own tooling and from the lag dashboard, then restore throughput by scaling the group up to the topic's partition count, and no further, because a fourth consumer gets no partition and adds nothing.
DataVerifiedk3dkind
Definition on GitHubWhat you'll do
- Diagnose a lagging consumer group: name the topic, the group and the per-partition backlog from kafka-consumer-groups.sh and from the lag dashboard
- Restore throughput by scaling the consumer group, and prove from Kafka's own lag telemetry that the backlog actually drained while producers stayed ramped
- Respect the partition ceiling: three partitions means at most three useful consumers, so scale to the ceiling rather than past it
Stages
- 1event-flow
The topic, a balanced producer/consumer pair, and the metrics that make lag visible
consumer-group-resetorders-topicproducer-consumerkafka-metricskafka-lag-dashboard - 2build-lag
Ramp producers to 3x rate so the single consumer falls permanently behind
ramp-producers
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressdata/kafkamonitoring/metricsmonitoring/grafana
Related scenarios
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
- Chaos EngineeringRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
- Cost & Capacity: Right-Sizing Over-Provisioned Resourcesgo-api arrives reserving 2000m CPU and 1Gi of memory - forty times what it uses. Drive real traffic, read the gap between reserved and used on the dashboard and in OpenCost, then right-size the requests. Grading has two sides: a request over the waste ceiling fails, and so does one under the peak the workload actually reached, because under-provisioning is how workloads get throttled and OOM-killed.