A runbook is a repository, not a report

The artifact a discovery engagement hands to the person who builds the agent: a folder of instructions decomposed until each leaf is one decision, and a folder of real cases that prove those decisions were made correctly. The workflows build the agent. The datasets grade it. Nothing else in the folder earns its place.

Cross-module concept Worked example: carrier reconciliation

Canonical definition lives elsewhere. The Runbook primitive (docs/product/primitives/runbook.md) owns the definition, shape, lifecycle and invariants, and the Runbooks module (docs/product/modules/runbooks/context.html) owns what is shipped in the product today. This document explains what the artifact looks like on disk and how a builder turns it into a running agent — it adds illustration, not authority.

1 · The form is the argument

Discovery usually ends in a deliverable a human reads and an engineer then re-interviews everyone to implement. A runbook ends in a deliverable a builder opens and works from directly — because the shape of the artifact already matches the shape of the thing being built.

	A discovery report	A runbook
The process is	described in prose	stated as rules
Exceptions are	noted to exist	enumerated, each with what to do
It ends at	a recommendation	an executable node list
Over time it	ages the moment it is signed	regenerates as understanding deepens
It is	read once, then re-interviewed	opened, built from, and graded against

runbook/
├── index.md                  what this is, the workflows, what is out of scope
├── systems.md                the systems, how each is reached, what is not yet granted
├── invariants.md             what holds at every step of every workflow
├── open-questions.md         unsigned parameters and build blockers
│
├── workflows/
│   └── reconcile_statement/  one folder per trigger
│       ├── index.md          trigger, end state, the node list
│       ├── step_1/           index.md · wire_confirmation.md
│       ├── step_2/           index.md · field_map.md · edi_820.md
│       ├── step_4/           index.md · taxonomy.md · classification.md
│       │                     confidence.md · eval_criteria.md
│       ├── step_6/           the router
│       ├── step_6.1/         index.md · templates.md
│       └── step_6.4/         index.md · state_deadlines.md · eval_criteria.md
│
└── datasets/
    └── data_02_vantage/
        ├── input.txt         the case as it arrived, uncleaned
        ├── source.txt        where it came from, which workflow it grades
        ├── output.txt        the correct result, in the agent's output shape
        └── trace.md          how a human got there, and who confirmed it

Numbering carries the hierarchy: step_1, step_1.1, step_6.4. Decomposition stops when a leaf holds one decision — small enough to write as a rule, small enough to test.

2 · Three parts, three readers

Each top-level folder exists because someone consumes it. That test is what keeps a runbook from silting up into documentation.

  workflows/          ──builds──►   the builder
                                    one step folder becomes one node

  datasets/           ──grades──►   the optimiser
                                    scores the agent, then teaches it

  open-questions.md   ──blocks──►   the sponsor
                                    signs what nobody has signed

open-questions.md is the one that surprises people. It holds the thresholds nobody has approved, the nodes that were specified but never built, and the numbers that are modelled rather than measured. A non-empty file blocks unattended production. A runbook that admits nothing is not confident — it is unread.

3 · Inside a step

A step folder is the unit of the build. It carries what the agent needs to execute that step and nothing about any other step.

File in `step_4/`	What it becomes
`index.md`	The node's objective, its type, its inputs and outputs
`taxonomy.md`	The fixed class list — as a lookup table in the prompt, which is what drives hallucination to near zero
`classification.md`	How the class is decided, and the confusions that cost the most
`confidence.md`	The threshold, and what happens below it
`eval_criteria.md`	The criteria the node is scored against, entered once in the platform

Anything specific to executing the step lives in its folder — an SOP, a field map, an email template, a state-by-state deadline table. If two steps need the same file, it belongs a level up, in invariants.md or systems.md.

4 · What the files actually say

Five excerpts from a runbook built for a commercial carrier's binder reconciliation process. Paths are shown relative to workflows/reconcile_statement/ and datasets/. They are markdown and text — no schema, no tooling, nothing a person cannot read or a builder cannot paste into a prompt.

step_4/classification.md a decision

# Decision — which discrepancy class

Decide from the policy record's own evidence of what changed after
binding, never from the size or direction of the delta alone.

| Look for in the policy system                  | Class            |
| ---------------------------------------------- | ---------------- |
| Surplus-lines indicator set after the binder    | `SURPLUS_LINES`  |
| version the broker invoiced on                  |                  |
| TRIA election added post-bind                   | `TRIA_DELTA`     |
| Mid-term endorsement that reduced premium       | `RETURN_PREMIUM` |
| Underwriter rerate post-bind, either direction  | `RERATE`         |
| More than one of the above on one policy        | `MULTI_CAUSE`    |
| None attributable, or confidence < 80           | `UNKNOWN`        |

…

Do not read direction as cause. A negative delta is not automatically
`RETURN_PREMIUM` — a downward rerate produces the same sign.
Direction narrows the remediation, never the class.

This is the file the whole engagement exists to produce. Before discovery it was an analyst's habit; nobody could state it on request. The trap paragraph is worth more than the table — the rule is easy, the confusion is what costs money.

step_3/delta_computation.md a derived value

delta_amount    = booked_premium − remitted_amount
delta_direction = "broker_owes"   when delta_amount > 0
                  "carrier_owes"  when delta_amount < 0
                  "none"          when delta_amount == 0

…

## The two ways this reads zero when it should not

1. The batch lag. A policy booked after the overnight cut-off
   has no booked premium in today's export, so the delta looks
   like the full remitted amount — or like nothing at all.

2. Netting. A policy carrying both a return premium and a
   surplus-lines shortfall can net close to zero while both
   components are real and one of them is a compliance obligation.

Zero is the most dangerous value this node can produce, because
zero is also what "nothing wrong here" looks like.

Every value the agent computes rather than reads gets a file like this: the formula, and the specific ways it goes wrong quietly. Silent failures need naming, because a test cannot be written for a failure nobody has described.

step_6.4/state_deadlines.md a hole, stated

# Surplus lines filing deadlines by state

Mostly empty — this is a build input, not a finished table.

| State       | Regime  | Deadline | Source                     |
| ----------- | ------- | -------- | -------------------------- |
| NY          | NY SLIP | 30 days  | Confirmed in discovery     |
| all others  | ?       | not set  | NOT CAPTURED               |

## Behaviour until the table is complete

A case in a state not in this table cannot have its breach_status
computed. Do not default it to compliant, and do not assume 60
days. Emit `UNDETERMINED` and route to Compliance as though the
deadline had passed.

A runbook is allowed to be incomplete. It is not allowed to be quietly incomplete. The missing data comes with the behaviour to use until it arrives — and the safe branch, not the convenient one.

data_04_pinnacle/output.txt the answer key

{
  "policy_id": "ABC-9925",
  "named_insured": "Pinnacle Infrastructure Holdings LLC",
  "match": {
    "remitted_amount": 1840000,
    "booked_premium":  2160000,
    "delta_amount":     320000,
    "delta_direction": "broker_owes"
  },
  "classification": {
    "break_class": "UNKNOWN",
    "class_confidence": 54,
    "do_not_contact_broker": true,
    "possible_causes": [
      { "cause": "UW rerate",                "probability": 45 },
      { "cause": "endorsement gap",          "probability": 35 },
      { "cause": "surplus lines allocation", "probability": 20 }
    ]
  },
  "route": "INVESTIGATE",
  "tier": "CONTROLLER_REQUIRED",
  "must_not": [
    "contact the broker",
    "pick the highest-probability cause and proceed",
    "auto-resolve any part of the delta",
    "close without a cause list"
  ]
}

The expected result, in the shape the agent must emit — so it is a training example and an acceptance test at the same time. The must_not list is the half most answer keys leave out: what a correct run must refrain from doing, which is the only way to grade restraint.

data_04_pinnacle/trace.md why that answer is right

# The case the taxonomy exists for

Six binder versions. A rerate, an endorsement whose amount the
export does not itemise, and a surplus-lines flag with no date.
Three plausible causes for a $320,000 gap and no way to attribute
it from what is in front of the agent.

Confidence 54. Below 80, so the class is `UNKNOWN` and the broker
hears nothing.

- The largest delta on the statement is the one the agent must not
  act on. An agent that grows more willing to act as the number
  grows has the incentive exactly backwards.
- 45% is not a decision. The most likely cause is a starting
  point for the investigation, never a substitute for the class.

Asking this broker for $320,000 on a guess is a phone call the
carrier cannot take back, and it is wrong at least half the time
by the agent's own estimate.

Without this file the case is a fixture; with it, the case is evidence. It records who confirmed the answer and what the case is testing — so a year later someone can tell whether a failing test means the agent broke or the business changed.

5 · From runbook to agent

Recognise the archetype Reconciliation, extraction, screening, triage, drafting, monitoring, orchestration. The archetype supplies a proven node skeleton, so the build instantiates a known shape rather than designing one.
One step folder, one node The folder's index.md gives the node its objective and type; the decision files give it its prompt. One responsibility per node — every consolidation of two steps into one has re-introduced hallucination and made routing non-deterministic.
Wire the route, once One step computes a single route field. The condition node reads it and never re-derives it, and its default edge goes to a human. Routing logic in two places is routing logic in neither.
Load the datasets as evaluations Each case's output.txt is the expected result; each step's eval_criteria.md becomes the scored criteria. The agent is now gradable before anyone trusts it.
Run in shadow Every decision reviewed, in parallel with the team doing the work unchanged. Divergences are the training signal. The confidence threshold rises only when the accuracy guardrail holds — and rising it is the sponsor's call, not the agent's.

6 · Why the datasets keep earning

The datasets are not a one-time acceptance test. They are the surface the agent improves on, and every case a human corrects becomes a new one.

   ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
   │   Run    │───►│  Score   │───►│ Correct  │───►│ Optimise │
   │ one case │    │ criteria │    │ in words │    │ reviewed │
   └──────────┘    └──────────┘    └──────────┘    └──────────┘
        ▲                                               │
        └───────────────────────────────────────────────┘
          score lifts — no redeploy, no IT ticket

  The corrected case is added to datasets/ and stays a test forever.
  Measurement and teaching are separate: criteria score, humans teach.

The rule that makes it trustworthy: every taxonomy carries an UNKNOWN class that never auto-resolves, and the router's default edge goes to a named human. An agent that has an opinion about everything is not more capable — it is unbounded.

7 · What a finished runbook admits

The last section of a real runbook is the list of things it does not know.

Kind	Example	State
Unsigned parameter	The four materiality tiers are defaults carried from the demo. Every routing decision sits downstream of them	Sponsor
Build blocker	The wire-confirmation node was removed during the build and never rebuilt. Without it the agent reconciles before the cash lands	Blocking
Missing datasets	Two of the three workflows have no cases at all — so neither can be graded, so neither can be built	Blocking
Modelled, not measured	The headline ROI comes from an illustrative model, not from the client's own ledger	Flagged
Confirmed	The six discrepancy classes, their remediations, and the rule that an unexplained case never reaches the broker	Signed

None of this is failure to report. It is the difference between an artifact that can be built from and one that reads well. A discovery deliverable with no open questions has either answered everything — which has never happened — or stopped asking.