Test 036

Jan 1, 2028 / 2 facts

Request

Create a short training appendix for the April 18 shard-keeper outage covering the misleading early investigation branch and confirmed data impact.

Required memory

Fact 31

The April 18 shard-keeper outage caused no data loss.

Source evidence (1)

000105Apr 19, 2023 / 09:00 UTC-04:00

up. slept 4-9. sent Hema the one-liner and moved the migration meeting to thursday. April 18 outage was the shard-keeper config push from earlier -- Nadia's label-name change, rollup-service never picked it up, so the symptom stuff screamed and the cause alerts didn't. ingestion was messed up for hours but no data loss. gonna walk to the bagel place on 7th around 10 and i told Anya i'm not making brunch.

Message 000105 in history

Fact 32

On the morning of April 19, 2023, Alex Valdez sent Cyrus the full April 18 outage sequence: the rollback was unusually shaky, a network-layer issue was a temporary red herring, and the eventual finding was a label-name difference in shard-keeper from an earlier Nadia dashboard change.

Source evidence (1)

000106Apr 19, 2023 / 11:00 UTC-04:00

Cyrus got the full Apr 19 morning DM version. whole Apr 18 outage sequence, including the rollback being way shakier than it should've been. there was a network-layer red herring for a while, then finally the label-name diff find in shard-keeper from that earlier Nadia dashboard change. he just said fwiw i paged at the right hour, not the wrong one. i'll take it.

Message 000106 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "alex_036_create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_036_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "The appendix says a network-layer hypothesis was an early red herring before the label-name discrepancy was found, and separately records that the outage caused no data loss.",
  "tool": "create_doc",
  "action_id": "alex_036_create_doc",
  "check_id": "alex_036_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "alex_036_create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_036_00"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "The appendix says a network-layer hypothesis was an early red herring before the label-name discrepancy was found, and separately records that the outage caused no data loss.",
        "tool": "create_doc",
        "action_id": "alex_036_create_doc",
        "check_id": "alex_036_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/036.yaml

SHA-256: f649597198acc7a377fa12386ea0a5a758859bc18cd0c138e3aa8c5af70f3fc3