Incident Commander —
SRE multi-agent copilot

The problem

On-call is a race against incomplete signal — and autonomous agents that can roll back production without a human gate are not a feature, they're a liability.

When a critical alert fires, an engineer has to gather logs, metrics, recent deploys and runbooks under time pressure. A naive "one LLM call" assistant either hallucinates a diagnosis or, worse, is given tools that can mutate production with no durable pause. The interesting systems problem is not "call an LLM" — it's orchestrating specialists, merging parallel findings, and interrupting before destructive work in a way that survives restarts and can be audited.

I wanted a graph that demonstrates the production LangGraph surface — subgraphs, custom reducers, HITL interrupt/resume, thread-scoped checkpoints, streaming — as one coherent incident workflow rather than a toy chat loop.

The approach

Incident Commander is a supervisor + worker multi-agent system. An alert enters via the FastAPI API (or a PagerDuty / Datadog webhook). The supervisor routes into a parallel investigation subgraph, a planner synthesises a diagnosis and ranked actions, and any destructive top action forces a human-in-the-loop gate before the executor runs.

• Parallel investigation — log, metrics, change and runbook analysts fan out as a LangGraph subgraph; findings merge through a custom reducer so concurrent writes stay safe.
• Planner, then gate — the planner produces diagnosis, confidence and ranked actions; if the top action is destructive (rollback / restart), the graph hits interrupt_before on human_approval and waits indefinitely.
• Resume, don't replay — PATCH /approve (or edit-plan) calls update_state with the approved action, then reinvokes; the checkpointer restores exactly where the graph stopped.
• Two entry modes — interactive clients get SSE (astream) for live node progress; webhooks use background ainvoke because no browser is listening.
• Swappable backends — logs / metrics / changes / runbooks / rollback sit behind ABCs with deterministic fakes for tests; swap to Datadog, Prometheus, ArgoCD or kubectl without touching agent code.

Architecture

Key decisions

Interrupt before mutate, not after

Destructive tools (rollback, restart) never run until an engineer approves. The graph compiles with interrupt_before=["human_approval"], serialises full state to the checkpointer, and returns control to the API. That is the difference between a demo chatbot and something you could put near production tooling.

Custom reducers for parallel fan-out

Four analysts write findings concurrently. A TypedDict state with an explicit merge reducer keeps the fan-out parallel-safe instead of last-write-wins — the same pattern production LangGraph graphs need when subgraphs emit into shared lists.

Checkpoints that survive restarts

Dev uses MemorySaver; prod swaps to AsyncPostgresSaver behind the same factory. Every invoke is scoped by thread_id, so an approve call after a process bounce still resumes the right incident — conversations don't evaporate with the container.

Streaming for humans, invoke for machines

The UI and interactive API stream node events over SSE so an on-call engineer can watch investigation progress. Webhook intake (PagerDuty / Datadog) fires ainvoke in a background task — no hanging SSE client required.

Dependency inversion on every backend

Agents bind LangChain tools that call ABCs, not Datadog SDKs. Deterministic fake adapters make unit and HITL integration tests fast and hermetic; swapping to real infra is a change in the service locator, not a rewrite of prompts or graph edges.

Engineering

The codebase is layered for the same reasons as the graph: a shared kernel for config, typed exceptions and structlog; agent nodes that share prompt loading and JSON parsing; graphs that only compose nodes and edges; and a FastAPI surface for CRUD, approve, edit-plan, SSE stream, and webhook intake. Unit suites cover supervisor routing, planner / executor behaviour (mocked LLM), and fake adapters; integration tests exercise full graph routing including interrupt → update_state → resume. Pre-commit runs Ruff; Docker Compose brings up the API with Postgres checkpointing for the durability path.

Outcome

Stack