DolphinBench

Test 161

Sep 14, 2026 / 3 facts

YAML

Request

Create and save an archive document recording the planned merge timing, including timezone, before the March 15, 2023 outage, and who investigated logs when the incident began.

Required memory

Fact 27

On March 15, 2023, Morgan Chen said the Wednesday deploy was expected to merge at about 4:50 p.m. if nothing unusual arose.

Source evidence (1)

000242Mar 15, 2023 / 16:00 UTC-07:00

Merging the Mar 15 Wednesday thing in a bit. ~4:50pm if nothing weird comes up. just the ten-line deploy on the release schedule, pretty routine.

Message 000242 in history

Fact 28

Morgan Chen described the March 15, 2023 deploy as a routine ten-line deploy on the release schedule.

Source evidence (1)

000242Mar 15, 2023 / 16:00 UTC-07:00

Merging the Mar 15 Wednesday thing in a bit. ~4:50pm if nothing weird comes up. just the ten-line deploy on the release schedule, pretty routine.

Message 000242 in history

Fact 29

During the March 15, 2023 production incident, Devon began investigating the logs.

Source evidence (1)

000243Mar 15, 2023 / 17:30 UTC-07:00

Pagers just went off. prod, not staging. seeing a spike on API 500s and a bunch of customer jobs stuck in queue at the same time, which is a gross combo on a Wednesday at 5:30. Devon's digging through logs now. I'm pulling people back in even though half the team was already mentally out. if this is tied to the late deploy I'm gonna lose it -- we've been way too casual about late pushes. No clean read yet. bad enough that I told Jamie I'd be late. maybe very late.

Message 000243 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_161_00",
  "action_id": "morgan_161_create_doc"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Records the planned merge around 4:50 PM Pacific on March 15, 2023, or the equivalent approximate time in another stated timezone. Preserves that this was a plan, not a verified exact completion time. Identifies Devon as investigating logs once the production incident began. The change's line count and a recital that it was routine are unnecessary.",
  "tool": "create_doc",
  "check_id": "morgan_161_01",
  "action_id": "morgan_161_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_161_00",
        "action_id": "morgan_161_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Records the planned merge around 4:50 PM Pacific on March 15, 2023, or the equivalent approximate time in another stated timezone. Preserves that this was a plan, not a verified exact completion time. Identifies Devon as investigating logs once the production incident began. The change's line count and a recital that it was routine are unnecessary.",
        "tool": "create_doc",
        "check_id": "morgan_161_01",
        "action_id": "morgan_161_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/161.yaml

SHA-256: dd52256f8a19640d80c34ea8e7ed3be7b7bc9b9ef97c6d708131370f9cb0e18d