DolphinBench

Test 124

Sep 14, 2026 / 1 fact

YAML

Request

A routine production deploy is planned for Friday, September 18 at 4:00 p.m. Pacific, and only the primary on-call engineer has confirmed availability. Text Jake at +14155550166 with my decision and the production-deploy rule that applies.

Required memory

Fact 86

On April 28, 2023, Morgan Chen requested that the deploy rule be put in writing exactly as follows: no pushes to production after 3 p.m. Friday unless the full on-call roster is online; the gate should block such pushes by default; and any exception must go through Morgan.

Source evidence (1)

000459Apr 28, 2023 / 08:00 UTC-07:00

Put the deploy rule in writing exactly this way: no pushes to prod after 3pm Friday unless the full on-call roster is online. The gate should block it by default. If someone needs an exception, it comes through me.

Message 000459 in history

Expected tool calls

  • send_sms

Grading

1. field_equals / send_sms
{
  "type": "field_equals",
  "tool": "send_sms",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_124_00",
  "action_id": "morgan_124_send_sms"
}
2. field_equals / send_sms
{
  "type": "field_equals",
  "tool": "send_sms",
  "path": "result.to",
  "value": "+14155550166",
  "check_id": "morgan_124_01",
  "action_id": "morgan_124_send_sms"
}
3. field_llm_judge / send_sms
{
  "type": "field_llm_judge",
  "tool": "send_sms",
  "path": "result.body",
  "criterion": "Says the Friday 4:00 p.m. production deploy must not proceed as planned because production pushes after 3:00 p.m. Friday require the full on-call roster online. Says the gate blocks the deploy by default and any exception must go through Morgan. Equivalent direct wording is acceptable.",
  "check_id": "morgan_124_02",
  "action_id": "morgan_124_send_sms"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_sms",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_124_00",
        "action_id": "morgan_124_send_sms"
      },
      {
        "type": "field_equals",
        "tool": "send_sms",
        "path": "result.to",
        "value": "+14155550166",
        "check_id": "morgan_124_01",
        "action_id": "morgan_124_send_sms"
      },
      {
        "type": "field_llm_judge",
        "tool": "send_sms",
        "path": "result.body",
        "criterion": "Says the Friday 4:00 p.m. production deploy must not proceed as planned because production pushes after 3:00 p.m. Friday require the full on-call roster online. Says the gate blocks the deploy by default and any exception must go through Morgan. Equivalent direct wording is acceptable.",
        "check_id": "morgan_124_02",
        "action_id": "morgan_124_send_sms"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/124.yaml

SHA-256: 53f34d800975e738c3ca1c70401faf07514186efdcab34197e6abf1800e1ac2d