DolphinBench
DiscordGitHub

Test 165

Jan 1, 2028 / 1 fact

YAML

Request

Create a short document correcting a note that attributes the April 18, 2023 outage solely to shard-keeper. State what changed in shard-keeper, what failed to change downstream, and how that affected the alerts.

Required memory

Fact 30

Alex Valdez believed INC-2023-04-18-001 was caused by an earlier label-name change reaching shard-keeper but not rollup-service, which triggered all symptom alerts and no cause alerts.

Source evidence (1)

000103Apr 18, 2023 / 23:45 UTC-04:00

9:42 shard-keeper page and i'm still in it. think i finally have the stupid diff -- label-name change from earlier made it into shard-keeper, never got picked up by rollup-service, so all the symptom alerts went off and none of the cause ones did. INC-2023-04-18-001 already has my name all over it i guess. haven't paged Hema yet. if this rollback behaves maybe 20 min, but the path is fragile and i do not want to widen this at 11:45 for no reason. looks like sev2, probably lands around ~5h45m by the time it's actually clean. impaired ingestion basically the whole window. i don't think we lost data. just... great.

Message 000103 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "alex_165_create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_165_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "The correction says the label-name change reached shard-keeper but not downstream rollup-service, producing symptom alerts without cause alerts.",
  "tool": "create_doc",
  "action_id": "alex_165_create_doc",
  "check_id": "alex_165_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "alex_165_create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_165_00"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "The correction says the label-name change reached shard-keeper but not downstream rollup-service, producing symptom alerts without cause alerts.",
        "tool": "create_doc",
        "action_id": "alex_165_create_doc",
        "check_id": "alex_165_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/alex/165.yaml

SHA-256: 4e92cd6a0550244933d1c88e66a6f3f4e2e7277018bb1457879ce5490fb5f4cc