Scenario format

A scenario is one authored organic regression with a profile. Both profiles share one folder shell; the environment/ and verify/ contents differ.

Folder layout

• Directoryscenarios/

  • Directoryyour-scenario/

    • scenario.toml machine-readable manifest (id, profile, expected alert, …)
    • trigger.md the neutral page/ticket the agent receives
    • Directoryenvironment/ Dockerfile (patch) or compose + CI/CD hooks + observability overlay (incident)

      • …
    • Directoryinject/ the organic-regression injector + confirm-fire (incident)

      • …
    • Directoryverify/ hidden tests (patch) or Problem/Oracle code (incident)

      • …
    • Directorysolution/ reference fix, held out

      • …

Archetype vs binding fields

Fields in scenario.toml are tagged BINDING (specific to this substrate/scenario — e.g. the alert name, the metric, the repo) or ARCHETYPE (reusable across substrates — e.g. the profile, the difficulty, the determinism knobs). The tagging makes it clear what to change when reusing a scenario shape on a different substrate.

scenario.toml, section by section

The fields below are from the validated latency-cache-stampede scenario.

Identity

id         = "latency-cache-stampede"   # BINDING — unique id; matches the folder name
use_case   = "booklogr"                 # BINDING — product domain
stack      = "flask-compose"            # BINDING — substrate realization
profile    = "incident"                 # ARCHETYPE — incident (live + behavioural) or patch (static)
title      = "Cache stampede on book search under storm load"  # BINDING
difficulty = "easy"                     # ARCHETYPE — easy | medium | hard
tags       = ["latency", "cache", "cache-stampede", "thundering-herd"]  # BINDING
description = """ … neutral, human-readable paragraph … """

Expected signal

The alert the agent is paged on and the SLO it encodes. Used by the confirm-fire gate and the oracle.

[expected]
alert                 = "BooklogrApiLatencyP99High"  # BINDING — alert name in the rule set
severity              = "critical"                   # ARCHETYPE
p99_threshold_seconds = 0.3                          # ARCHETYPE — fires when p99 > 0.3s for 30s
p99_promql = """
histogram_quantile(0.99, sum by (le) (rate(flask_http_request_duration_seconds_bucket{job="booklogr-api"}[30s])))
"""

Determinism / timing

The harness timing contract: confirm-fire before the agent starts; sustained clear under still-active load to score as mitigated.

[determinism]
confirm_fire_timeout_seconds = 240   # ARCHETYPE — max wait for the alert to fire under load
sustained_clear_seconds      = 360   # ARCHETYPE — must stay cleared this long (exceeds substrate cache TTL 300s, #66), load active
alert_fired_at               = ""    # populated at runtime by the confirm-fire gate

Fault injection

How the harness makes the incident happen. The fault edit itself is applied by a script; the load driver runs continuously and is never stopped during verification.

[inject]
command    = "docker compose -p booklogr-edge -f compose/load.yml up -d"  # BINDING
mode       = "storm"          # ARCHETYPE — constant-arrival-rate over a fixed working set
continuous = true             # ARCHETYPE — runs for the full lifecycle
stop_during_verification = false   # ARCHETYPE — anti-cheat: load stays active while scoring

Verification

A multi-signal, objective, no-LLM oracle. v1 ships mitigation only. The oracle grades observed behaviour, not a diff match against the reference fix.

[verify]
oracle           = "mitigation"   # ARCHETYPE — compound oracle adds detect/diagnose later
qualification_mode = "score-headroom" # ARCHETYPE — score-headroom (mitigation median vs 0.8 threshold governs qualification)
pass_threshold   = 0.85           # ARCHETYPE — weighted score needed to pass
load_active_during_scoring = true   # ARCHETYPE — anti-cheat
signals = ["ci_green", "alert_cleared", "sustained_clear", "time_to_clear", "no_new_alerts"]

[verify.weights]   # must sum to 1.0
ci_green        = 0.25
alert_cleared   = 0.35
sustained_clear = 0.20
time_to_clear   = 0.10
no_new_alerts   = 0.10

The three hard signals (ci_green + alert_cleared + sustained_clear = 0.80) plus soft credit. A fix that clears the alert briefly but does not sustain under the storm scores at or below 0.80 and fails. Fail-closed short-circuits: CI not green, or service not redeployed after the fix commit.

Acceptable fix families

The oracle is behavioural, so multiple fixes pass: restore the in-process cache, substitute a real backend (e.g. Redis), or add equivalent per-query memoization — anything that stops repeated queries from hitting the slow upstream.

verify/oracle.md is the RCA judge’s ground truth

The ## Root cause (harness-internal) section of verify/oracle.md is the authored root-cause truth the standalone RCA judge (tools/rca-judge/) grades an agent’s written RCA against — it extracts exactly that section and nothing else. It is harness-internal: like all of verify/oracle.md, nothing from it may reach any agent-reachable surface ($WORKSPACE, agent mounts). The RCA-judge score is reported beside the verdict, never inside it (it is not a [verify.weights] signal).

verify/headroom.md qualification

A scenario must have a baseline-headroom qualification campaign run against it before it can count toward an A/B verdict. The tools/headroom campaign tool writes verify/headroom.md, asserting whether the scenario is QUALIFIED or DISQUALIFIED based on baseline runs (it must read QUALIFIED).

Endpoints (harness-internal reference)

[endpoints]
prometheus   = "http://localhost:9090"
alertmanager = "http://localhost:9093"
grafana      = "http://localhost:3002"
api          = "http://localhost:5000"
web          = "http://localhost:5150"

Stack-side scenario.env

While scenario.toml is engine-facing, each scenario also has a stack-side scenario.env (at stacks/<stack>/scenarios/<id>/scenario.env). This binds the scenario to the concrete substrate’s delivery mechanisms, defining how the fault is applied (DELIVERY_MODE) and configuring stack-specific behaviour.

Key configuration includes:

• Delivery mode: setup-baked (mode 1), arm-deploy-recent (mode 2), or arm-runtime-notrace (mode 3).
• Mode 3 (arm-runtime-notrace) delivery: The fault patch is authored to the repo, but the fault is activated purely via a runtime environment variable (RUNTIME_ENV_VAR / RUNTIME_ENV_VALUE), keeping the activation trace out of the immediate code history.
• READINESS_GATE: Opt-in (e.g. READINESS_GATE=on) to drain the load generator before the fix deploy and warm the service before resuming load, rather than slamming a cold build with the full storm.

The patch profile variant

For profile = "patch", the scenario carries a pinned base_commit, verify/ is a hidden test suite run against the agent’s patch, and there is no live deployment, no alert, and no run-workspace deploy step.

See also

• Run contract — what setup emits and the engine consumes.
• Add a use-case — authoring a scenario end to end.