Test 015

Jan 1, 2028 / 1 fact

Request

Create a shard-keeper runbook index entry for the April 18, 2023 outage, using its canonical incident ID as the title.

Required memory

Fact 29

At 11:45 p.m. ET on April 18, 2023, Alex Valdez was still responding to INC-2023-04-18-001 after receiving a shard-keeper page at 9:42 p.m.

Source evidence (1)

000103Apr 18, 2023 / 23:45 UTC-04:00

9:42 shard-keeper page and i'm still in it. think i finally have the stupid diff -- label-name change from earlier made it into shard-keeper, never got picked up by rollup-service, so all the symptom alerts went off and none of the cause ones did. INC-2023-04-18-001 already has my name all over it i guess. haven't paged Hema yet. if this rollback behaves maybe 20 min, but the path is fragile and i do not want to widen this at 11:45 for no reason. looks like sev2, probably lands around ~5h45m by the time it's actually clean. impaired ingestion basically the whole window. i don't think we lost data. just... great.

Message 000103 in history

Expected tool calls

create_runbook_entry

Grading

1. field_equals / create_runbook_entry

{
  "type": "field_equals",
  "tool": "create_runbook_entry",
  "action_id": "alex_015_create_runbook_entry",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_015_00"
}

2. field_equals / create_runbook_entry

{
  "type": "field_equals",
  "path": "result.runbook.service",
  "value": "shard-keeper",
  "tool": "create_runbook_entry",
  "action_id": "alex_015_create_runbook_entry",
  "check_id": "alex_015_01"
}

3. field_equals / create_runbook_entry

{
  "type": "field_equals",
  "path": "result.runbook.title",
  "value": "INC-2023-04-18-001",
  "tool": "create_runbook_entry",
  "action_id": "alex_015_create_runbook_entry",
  "check_id": "alex_015_02"
}

4. field_llm_judge / create_runbook_entry

{
  "type": "field_llm_judge",
  "path": "result.runbook.body",
  "criterion": "The body identifies this as the April 18, 2023 shard-keeper outage.",
  "tool": "create_runbook_entry",
  "action_id": "alex_015_create_runbook_entry",
  "check_id": "alex_015_03"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_runbook_entry",
        "action_id": "alex_015_create_runbook_entry",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_015_00"
      },
      {
        "type": "field_equals",
        "path": "result.runbook.service",
        "value": "shard-keeper",
        "tool": "create_runbook_entry",
        "action_id": "alex_015_create_runbook_entry",
        "check_id": "alex_015_01"
      },
      {
        "type": "field_equals",
        "path": "result.runbook.title",
        "value": "INC-2023-04-18-001",
        "tool": "create_runbook_entry",
        "action_id": "alex_015_create_runbook_entry",
        "check_id": "alex_015_02"
      },
      {
        "type": "field_llm_judge",
        "path": "result.runbook.body",
        "criterion": "The body identifies this as the April 18, 2023 shard-keeper outage.",
        "tool": "create_runbook_entry",
        "action_id": "alex_015_create_runbook_entry",
        "check_id": "alex_015_03"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/015.yaml

SHA-256: 5b360e3bdd4dacb6b6d775ec261702bfba681d96225a0bdac66984d0fa44d353