Test 148 for Morgan Chen

Request

Create an archive note explaining the mechanism of the March 15, 2023 production outage and why its impact was uneven across tenants.

Required memory

Fact 30

On March 15 at 9 p.m. during the outage, Morgan Chen documented the reproduction path: a request on the new path encountered a cache miss, fell through to the old entitlement check, retried against the worker pool, and piled up there.

Source evidence (1)

000244Mar 15, 2023 / 21:00 UTC-07:00

Mar 15, 9pm, still in the outage. trying to write this down clean while they're heads down: repro is request comes in on the new path, cache miss, falls through to the old entitlement check, then retries against the worker pool and piles up there. the ten-line change looked harmless bc diff was tiny, but the asymmetric blast radius is real -- only some tenants hit the bad branch, and when they do it fans out 1 request into like 6-8 internal calls. so dashboards looked half-normal for too long. Devon is on traces, Priya rolled traffic back from the edge flag but we still have stragglers bc jobs already queued. note for the retro: no more shipping branchy auth/entitlement stuff behind a flag without a replay against prod-like traffic. tiny diff, huge mess.

Message 000244 in history

Fact 31

Morgan Chen reported that the ten-line change had an asymmetric blast radius: only some tenants reached the bad branch, where one request fanned out into roughly 6–8 internal calls, causing dashboards to look half-normal for too long.

Source evidence (1)

000244Mar 15, 2023 / 21:00 UTC-07:00

Mar 15, 9pm, still in the outage. trying to write this down clean while they're heads down: repro is request comes in on the new path, cache miss, falls through to the old entitlement check, then retries against the worker pool and piles up there. the ten-line change looked harmless bc diff was tiny, but the asymmetric blast radius is real -- only some tenants hit the bad branch, and when they do it fans out 1 request into like 6-8 internal calls. so dashboards looked half-normal for too long. Devon is on traces, Priya rolled traffic back from the edge flag but we still have stragglers bc jobs already queued. note for the retro: no more shipping branchy auth/entitlement stuff behind a flag without a replay against prod-like traffic. tiny diff, huge mess.

Message 000244 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_148_00",
  "action_id": "morgan_148_create_doc"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Traces the new request path through a cache miss into the old entitlement check, followed by retries piling up in the worker pool. Only some tenants hit the bad branch, where one request produced roughly 6-8 internal calls. No separate retrospective agenda is required.",
  "tool": "create_doc",
  "check_id": "morgan_148_01",
  "action_id": "morgan_148_create_doc"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_148_00",
        "action_id": "morgan_148_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Traces the new request path through a cache miss into the old entitlement check, followed by retries piling up in the worker pool. Only some tenants hit the bad branch, where one request produced roughly 6-8 internal calls. No separate retrospective agenda is required.",
        "tool": "create_doc",
        "check_id": "morgan_148_01",
        "action_id": "morgan_148_create_doc"
      }
    ]
  }
}