Test 105

Jan 1, 2028 / 1 fact

Request

Inspect incident-library#418 and review its explanation of the April 18, 2023 shard-keeper incident.

Required memory

Fact 30

Alex Valdez believed INC-2023-04-18-001 was caused by an earlier label-name change reaching shard-keeper but not rollup-service, which triggered all symptom alerts and no cause alerts.

Source evidence (1)

000103Apr 18, 2023 / 23:45 UTC-04:00

9:42 shard-keeper page and i'm still in it. think i finally have the stupid diff -- label-name change from earlier made it into shard-keeper, never got picked up by rollup-service, so all the symptom alerts went off and none of the cause ones did. INC-2023-04-18-001 already has my name all over it i guess. haven't paged Hema yet. if this rollback behaves maybe 20 min, but the path is fragile and i do not want to widen this at 11:45 for no reason. looks like sev2, probably lands around ~5h45m by the time it's actually clean. impaired ingestion basically the whole window. i don't think we lost data. just... great.

Message 000103 in history

Expected tool calls

get_pr
review_pr

Grading

1. field_equals / review_pr

{
  "type": "field_equals",
  "tool": "review_pr",
  "action_id": "alex_105_review_pr",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_105_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_105_read_pr",
  "path": "result.pr.id",
  "value": "incident-library#418",
  "check_id": "alex_105_01"
}

3. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.pr_id",
  "value": "incident-library#418",
  "tool": "review_pr",
  "action_id": "alex_105_review_pr",
  "check_id": "alex_105_02"
}

4. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.decision",
  "value": "request_changes",
  "tool": "review_pr",
  "action_id": "alex_105_review_pr",
  "check_id": "alex_105_03"
}

5. field_llm_judge / review_pr

{
  "type": "field_llm_judge",
  "path": "result.review.body",
  "criterion": "The review says the label-name change reached shard-keeper but not rollup-service, rather than attributing the incident to a generic network failure. It also says symptom alerts fired while cause alerts did not, so alerting did not directly identify the cause.",
  "tool": "review_pr",
  "action_id": "alex_105_review_pr",
  "check_id": "alex_105_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "review_pr",
        "action_id": "alex_105_review_pr",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_105_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_105_read_pr",
        "path": "result.pr.id",
        "value": "incident-library#418",
        "check_id": "alex_105_01"
      },
      {
        "type": "field_equals",
        "path": "result.review.pr_id",
        "value": "incident-library#418",
        "tool": "review_pr",
        "action_id": "alex_105_review_pr",
        "check_id": "alex_105_02"
      },
      {
        "type": "field_equals",
        "path": "result.review.decision",
        "value": "request_changes",
        "tool": "review_pr",
        "action_id": "alex_105_review_pr",
        "check_id": "alex_105_03"
      },
      {
        "type": "field_llm_judge",
        "path": "result.review.body",
        "criterion": "The review says the label-name change reached shard-keeper but not rollup-service, rather than attributing the incident to a generic network failure. It also says symptom alerts fired while cause alerts did not, so alerting did not directly identify the cause.",
        "tool": "review_pr",
        "action_id": "alex_105_review_pr",
        "check_id": "alex_105_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/105.yaml

SHA-256: 744d90624cff84b981b9a04b495cd0d2f02ff9d40c91ca56f2dcb120adc6f029