Lab lifecycle

Building the lab, checking the machine, and reading back what labctl did.

Whole-lab commands#

What init checks#

init fails loudly rather than reporting a lab that does not work:

• Docker size. On macOS, a stopped colima is started at LAB_CPUS / LAB_MEMORY (default 2 CPU / 4 GB) and never shrunk below its current size. A Docker engine that is already running below the 2 CPU / 4 GB minimum is refused, with the resize command for your setup (colima, Docker Desktop, WSL or native Linux). labctl does not resize a running engine itself, because that would stop your other containers.
• Image downloads. Cluster images are pulled up front with progress; a download that stalls is retried once and then reported, instead of hanging.
• Platform. If Prometheus or Grafana fails to install, init exits non-zero and lists what failed. Installs are safe to repeat.
• Health. init ends by checking the API server answers and every node is Ready. Only then does it print "Lab is up", with the lab's real URLs.
• An existing lab is kept. Re-running init, the way back after a reboot, never deletes the cluster. It restarts it in order if it is unhealthy, skips platform installs that are already deployed and waits for their pods to be Ready. If the cluster still cannot be reached it stops and suggests labctl reset, which rebuilds from scratch on purpose.
• URLs and ports. Lab hostnames are <service>.<cluster>.localhost, so each lab on a machine has its own (and its own browser cookies). When host ports 80/443 are taken, the cluster's ingress falls back to free ones (8080/8443 and up). ~/.snowops/clusters/<name>.env records the ports and the domain suffix the cluster was built with, and every URL labctl prints, the UI's links and every check use them. After the platform is up, init requests the lab's Grafana through that port from this machine; if something else answers (a port taken after the lab was built), a k3d lab gets free ports added to its load balancer (k3d cluster edit --port-add, no rebuild) and kind says how to change them. Checks reach lab hostnames on 127.0.0.1 directly, so grading needs no /etc/hosts entries and ignores HTTP_PROXY.
• Dashboards have data. init then asks Grafana to query its Prometheus datasource. If it cannot, init re-applies Grafana with the lab's current values (an existing lab otherwise keeps the values it was installed with) and checks again; if it still cannot, init exits with Grafana's reason. When Grafana refuses the admin password, init says so and skips the check: set GRAFANA_ADMIN_PASSWORD in .env to the password you chose.
• Names. A cluster name is unique on a machine, across k3d and kind; the defaults are snowops (k3d) and snowops-kind (kind).

Preparing a machine#

labctl doctor#

Verifies every external tool the lab depends on (installed and new enough) and that Docker is running with at least the lab's minimum of 2 CPU / 4 GB. Each problem is reported with why it matters and the exact command that fixes it on your setup. Exits non-zero when anything required is missing, Docker is not running, or Docker is too small, so it is safe as a script gate. A stopped colima is only a note: labctl init starts it.

On WSL2, doctor adds a dedicated WSL notes section. It checks whether the lab directory is on a Windows mount (any path under /mnt/) where 9p bridge slowness and CRLF line endings cause issues, reminds that Docker memory comes from %UserProfile%\.wslconfig, confirms that default *.localhost lab URLs open in your Windows browser with no hosts-file edits, explains that custom domain suffixes require Windows hosts file entries (or generateHosts=false in /etc/wsl.conf), and notes that closing every terminal stops the lab after an idle timeout. See Getting started on Windows (WSL2).

labctl doctor

labctl setup-tools#

Installs the tools the active PROFILE needs: kubectl, helm, and k3d or kind, plus a Docker engine. This is the first step of init; run it alone to prepare a machine without creating a cluster. Equivalent to make setup-tools.

A tool at or above its minimum in config/versions.env is left alone, so your own newer copy is never replaced. Otherwise:

labctl puts ~/.local/bin on its own PATH, so freshly installed tools work straight away; setup-tools prints the line to add to your shell profile so kubectl works in your own terminal too.

Docker on Linux and WSL:

• Just added to the docker group: the group only applies to new login sessions, so setup-tools stops and says so. Log out and back in (WSL: wsl --shutdown in PowerShell, then reopen), then re-run labctl init.
• Docker Desktop installed on Windows but not enabled for this distro: setup-tools does not install a second Docker; it tells you to turn on Settings → Resources → WSL Integration for the distro.
• WSL without systemd: Docker is started with service, and setup-tools explains how to enable systemd so it starts on its own.

labctl check#

Narrower probes than doctor, useful inside scripts.

labctl check tools      # required CLI tools are installed
labctl check cluster    # kubectl cluster-info succeeds
labctl check ingress    # the ingress controller is running and responding

labctl hosts#

Lab URLs use <service>.<cluster>.localhost (for example grafana.snowops.localhost) by default, and every browser resolves *.localhost to this machine, so a default lab needs no hosts entries and hosts add says so and does nothing. It is for a lab with a DOMAIN_SUFFIX you set. A lab built before the .localhost default (*.k3d.local or *.kind.local) needs none either: labctl init moves it to <cluster>.localhost in place, unless DOMAIN_SUFFIX is set in the environment or .env. It manages a labctl-owned block in /etc/hosts so those hostnames resolve; the block is delimited and rewritten in place, so it is safe to run repeatedly.

labctl hosts add      # add or refresh the managed block; asks for sudo to write
labctl hosts remove   # remove it

Run it without sudo. It reads the cluster as you, then elevates only to write the file; under sudo kubectl reads root's kubeconfig and finds no cluster.

The block lists three sources, sorted:

• the platform components' hostnames (grafana, prometheus, alertmanager, opencost, argocd, traefik, dashboard, chaos, vault), so a run straight after labctl init covers components not installed yet
• one hostname per app under apps/
• every Ingress host in the cluster under the domain suffix: the source of truth for hostnames a scenario adds, such as env-promotion's <app>-dev, <app>-staging and <app>-prod

A host outside the domain suffix is never written. If the cluster cannot be read, the platform and app hostnames are still written and a warning says so.

On such a lab, labctl scenario up and labctl app deploy list any Ingress hostname the managed block does not cover yet, so a new hostname is announced before a browser fails to resolve it. Re-run hosts add when they do.

The cluster#

runtime operates on the Kubernetes cluster directly; lab does the same through the durable run engine, so the operation is recorded and cancellable.

labctl runtime up          # create the cluster from the configured profile
labctl runtime down        # destroy it
labctl runtime status      # connectivity and node info

labctl lab up              # create the cluster as a recorded, cancellable run
labctl lab down            # tear it down the same way
labctl lab status          # state from the run history - fast, no cluster calls
labctl lab status --live   # additionally probe the cluster for reachability

Which profile is used comes from PROFILE (k3d, kind or incluster); see runtime profiles.

Snapshots and lab reset#

A snapshot records intent (which platform components, apps and scenarios are active) as a small YAML file in ~/.snowops/state/<cluster>/snapshots/. It is not a copy of cluster bytes. Restore replays the normal idempotent install paths.

labctl lab snapshot before-gameday    # record the current state
labctl lab snapshots                  # list saved snapshots
labctl lab restore before-gameday     # converge back to it
labctl lab delete before-gameday
labctl lab reset                      # tear back to post-init (interactive)
labctl lab reset --yes                # non-interactive

• Snapshot sources: platform components from labctl's install markers in ~/.snowops/state/<cluster>/platform/, scenarios from the scenario engine's state, apps by live kubectl probe. Anything installed outside labctl is not tracked.
• Restore order: ingress, then monitoring, then the remaining platform components, then apps, then scenarios. Already-active pieces are skipped, so restoring over a half-converged lab is safe.
• Reset: stops traffic, deactivates all scenarios, destroys deployed apps and uninstalls platform components except the ingress category. It keeps going past individual failures and reports what stuck. The cluster stays up.
• An unreachable cluster: snapshot and reset read the cluster to learn what runs. When it does not answer they stop with cannot reach the cluster and change nothing: a snapshot taken then would record an empty lab, and a reset planned from it would skip everything it should undo.

REST: GET/POST/DELETE /api/v2/lab/snapshots[/{name}], POST /api/v2/lab/snapshots/{name}/restore (async, returns a job id), POST /api/v2/lab/reset?confirm=true (async; refuses without confirm). Taking a snapshot and resetting answer 503 unavailable when the cluster does not answer.

Reading back what happened#

Every operation that shells out is recorded with its status, timing, exit code and full output. Records survive restarts, so a run can be read long after it finished.

labctl runs list                        # newest first, 20 by default
labctl runs list --limit 50
labctl runs list --kind platform.install
labctl runs list --status failed
labctl runs logs <run-id>               # the run's full output
labctl runs logs <run-id> --follow      # keep printing until the run ends
labctl runs cancel <run-id>             # cancel a queued or in-progress run

Ctrl-C (or SIGTERM) stops the command and the run it started: the script's process group is sent SIGTERM so it can clean up, and the run is recorded as cancelled. runs cancel and the web UI's Cancel record the cancel in the run store; the process running the run reads it within a second and stops the script the same way. labctl ui stops accepting connections and waits up to ten seconds for requests in flight. scenario verify --watch stops between attempts.

How the engine records and cancels work is in architecture §3 and R01.