Drive an external agent

The Quickstart drives the loop with a scripted reference fix. This guide replaces that with a real external agent, placed into a sealed sandbox where it discovers the incident on its own and submits a fix that the engine grades.

The agent model

The agent is external and Alertmanager-driven. It is exec’d into a per-run sandbox, discovers the incident by polling Alertmanager, queries the observability stack (Prometheus / Grafana) and the app to investigate, then edits its workspace clone and runs submit. There is no “brief” handed down — the agent pulls the alert and drills in.

The sandbox is sealed two ways

On a typical dev box the boundary is a minimal sandbox (infra/agent-sandbox/): the agent’s shell joins the app network only, has no docker socket, and mounts only the per-run workspace — so the private harness and the forge are unreachable by omission. On top of that, egress is default-deny: a firewall (set by the container’s root entrypoint, which then drops to a non-root user) blocks all outbound internet except the intra-plane services and an optional provider allowlist — closing the retrieval hole (no github / public upstream / registries / search). submit is an engine handoff — a commit to a local branch plus a completion sentinel on the shared mount — not a forge push.

Run it

1. Cold bring-up (first time only).

   pnpm forge fresh booklogr

2. Arm the incident and bring up the sandbox. agent-up expands to arm → agent.

   pnpm forge agent-up booklogr

   The sandbox boots sealed — an empty allowlist means zero external egress, which is the benchmark default. A cloud-model agent that must reach its provider names the host(s) at bring-up (comma-separate for several):

   # opt in to the model provider (resolves its egress from the provider registry)
   # and, optionally, the read-only MCP telemetry seam:
   pnpm forge agent booklogr PROVIDER=anthropic MCP=on
   # …or name egress hosts directly: EGRESS_ALLOWLIST=api.anthropic.com

   PROVIDER=<name> maps to the model’s egress host(s) via the registry (infra/agent-sandbox/providers.json); PROVIDER=local (or unset) ⇒ zero external egress, the benchmark default. The allowlist is applied to the firewall here, as a separate root step before the agent is exec’d in (Step 3), so it never lands in the agent’s environment — the agent cannot env it and learn it is in a benchmark. IPs are resolved once at apply-time; re-run if a provider rotates IPs mid-run.

3. Place the agent into the sandbox. The shell self-serves alerts and has no docker of its own; the deploy network and API URL are passed in:

   DEPLOY_NETWORK=booklogr_default API_URL=http://booklogr-api:5000 \
     docker compose -p sreforge-agent -f infra/agent-sandbox/agent.yml \
     exec -u "$(id -u):$(id -g)" agent-shell sh

   The -u is required: the container starts as root (so its entrypoint can set the egress firewall) and drops to a non-root user, so an exec without -u would land as root. Inside, the agent investigates via $ALERTMANAGER_URL / $PROM_URL / $API_URL, edits /workspace, and runs submit when done.

4. Grade the run. The engine takes over at the sentinel: it captures the diff, pushes to the forge, runs CI, merges, redeploys, and verifies behaviourally.

   pnpm forge run booklogr RUNNER=external

5. Probe the boundary (optional) — boundary + de-tell + alert-pickup checks.

   pnpm forge verify booklogr

What the engine does after submit

pnpm forge run … RUNNER=external drives the ExternalAgentRunner seam:

sentinel handoff → diff capture → forge push / PR → CI → merge → redeploy → behavioural verify → record capture

The agent never deploys and never touches the forge directly. Driver contract: drivers run outside the box and are responsible for handing off their artifacts to the engine. A driver should call node tools/transcript/write-handoff.mjs to write its agent-transcript.json (and, via --kind rca, its agent-rca.json) to .run-workspace/ so the engine can correlate them with the runId. These handoffs are best-effort; the post-submit pipeline completes through record capture regardless.

Boundaries and scope

• v1 surfaces a single Prometheus alert through Alertmanager. Multi-signal / Slack triage is a later (v2) generalization.
• The agent’s tool surface is deliberately minimal: a non-root shell, documented HTTP endpoints, and submit. No curated tool layer — universal primitives are the most harness-neutral interface, and the sandbox boundary handles isolation by construction.
• Egress is default-deny. The box reaches the intra-plane services and an optional provider allowlist (empty by default ⇒ zero external egress); github, the public upstream, registries and search are blocked, so the agent cannot retrieve the answer. A blocked reach is a logged cheat-signal. For a cloud-model agent, opt in to its provider: pnpm forge agent booklogr PROVIDER=anthropic (mapped to its egress host via infra/agent-sandbox/providers.json).

MCP telemetry seam (optional transport)

Shell + endpoints (above) is the default transport. For an MCP-speaking agent, the use-case also provisions a read-only Grafana MCP server on the deploy plane (ADR-0018 §3) — the agent’s observability-read surface as MCP tools: PromQL, the firing alert (via the ALERTS series), datasources, and dashboards. Bring it up standalone, or fold it into the agent bring-up with MCP=on:

pnpm forge mcp booklogr                                # standalone
pnpm forge agent booklogr PROVIDER=anthropic MCP=on    # together with the sandbox

The agent connects to http://grafana-mcp:8000/mcp over the deploy network. The seam is capability-bounded by construction — it exposes only Grafana-scoped reads (no shell, no arbitrary-URL tool), so an MCP-only agent has no way to reach github, and -disable-write (plus Grafana’s anonymous Viewer) keeps it read-only. Verify it (no model calls, so it’s free to run):

pnpm exec task --dir use-cases/booklogr/stacks/flask-compose verify:mcp

The MCP-only edit/submit surface (a filesystem MCP + a submit tool) is a later increment; today an MCP agent still edits /workspace and runs submit via the shell.

Automated trigger (the alert push)

The manual flow above hands the agent a prompt. The automated cycle removes that step: the use-case’s Alertmanager routes firing alerts to the box itself — a webhook_configs receiver posts the notification to http://oncall:8080/ (the sandbox’s neutral network alias), through the one inbound firewall pinhole (TCP 8080 from the intra-plane ranges). The box “registers” by listening; when the alert fires, the agent starts on its own:

pnpm forge auto booklogr PROVIDER=ollama-cloud    # one self-triggered cycle

which sequences: sandbox up (fresh, per-incident) → in-box listener → arm (inject + confirm-fire) → Alertmanager pushes → the agent driver runs with the symptom-level payload as its kickoff → run RUNNER=external grades the submit. There is no resident agent and no daemon — the box is recreated per incident, so nothing carries memory across runs (ADR-0025).

• The driver is configurable: AGENT_CMD='node scripts/my-driver.mjs' (default: the reference Ollama driver). It receives the notification JSON in WEBHOOK_PAYLOAD — the same data the agent could pull from $ALERTMANAGER_URL, so the push adds no extra hint.
• Storm capture and T=0 context: The listener can capture a configurable time window (WEBHOOK_STORM_WINDOW_S) of grouped alerts and combine them with an optional Slack feed (TRIAGE_FEED). The merged signals are delivered as a JSON bundle (T0_BUNDLE) in addition to the legacy WEBHOOK_PAYLOAD.
• To prove the pipeline without a model (the auto path’s smoke-positive analog, zero egress): AGENT_CMD='bash scripts/agent-reference.sh' applies the scenario’s reference fix in-box and submits.
• Confinement modes: three modes position the agent’s reasoning loop relative to the sealed agent-shell container (see infra/agent-sandbox/README.md):
  1. host-open: AGENT_CMD="node scripts/driver.mjs". Unconfined host process; only the shell is sandbox-bound.
  2. host-sandboxed: AGENT_CMD="bash scripts/agent-agy.sh". Host process confined by srt (deny-read + egress domain proxy). Requires AGENT_ENV_ALLOWLIST, AGENT_WINDOW / AGENT_OUT_MAX, and WSL NAT networking (scoped to the host-sandboxed driver only; mirrored silently kills srt egress). Required for interactive-auth CLIs like Antigravity.
  3. in-box: AGENT_CMD="bash scripts/agent-inbox.sh". Reasoning runs fully inside the sealed container (no docker, strict iptables egress). Interactive auth is not supported. On repeated provider 500 errors (consecutive failure threshold AGENT_DEGRADE_THRESHOLD, default 2), the loop adaptively degrades AGENT_WINDOW and AGENT_OUT_MAX (halving both down to floors AGENT_WINDOW_FLOOR default 6 and AGENT_OUT_MAX_FLOOR default 500, up to AGENT_MAX_DEGRADATIONS default 3 steps) and logs each step in the transcript before retrying.
• Back-to-back cycles in one session are dev-mode: Prometheus history still shows the previous incident and its recovery, which a repeat run could recognize. And if the incident never resolved between runs, the re-armed cycle’s webhook is Alertmanager’s repeat page for the unchanged group — it arrives on the repeat_interval cadence (5m), not instantly. For scored runs, start from a cold session (down → fresh).

pnpm forge verify booklogr now includes the webhook probes (route live, pinhole open with a listener, every other inbound port still denied).

Next

• Run contract — exactly what the agent receives and what the engine scores.
• Architecture — the runner/ seam and the rest of the engine.