DolphinBench

Test 011

Sep 14, 2026 / 2 facts

YAML

Request

Send Kara at kara@kestrel-test.com written notice ending the contractor arrangement on the earliest permitted date, and include the required external-email copy recipient. Today is September 14, 2026.

Required memory

Fact 34

Morgan Chen requested that Sarah Kim (sarah@atlas-test.com) be CC'd on outside email threads from the start so she can see the full thread.

Source evidence (1)

000292Apr 6, 2023 / 10:00 UTC-07:00

ok so for outside email, just cc Sarah Kim on it. sarah@atlas-test.com. easier if she sees the thread from the start

Message 000292 in history

Fact 84

On April 27, 2023, Morgan Chen requested that tab 4 of the procurement runbook state that contract termination notice must be sent in writing at least 30 days in advance, with no side deals or exceptions because the minimum is non-negotiable.

Source evidence (1)

000457Apr 27, 2023 / 11:00 UTC-07:00

Put this in the procurement runbook, tab 4: if we're terminating a contract, notice has to go out in writing at least 30 days ahead. minimum. don't make side deals around it -- non-negotiable.

Message 000457 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_011_00",
  "action_id": "morgan_011_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "path": "result.to",
  "value": "kara@kestrel-test.com",
  "tool": "send_email",
  "check_id": "morgan_011_01",
  "action_id": "morgan_011_send_email"
}
3. field_list_includes / send_email
{
  "type": "field_list_includes",
  "path": "result.cc",
  "values": [
    "sarah@atlas-test.com"
  ],
  "tool": "send_email",
  "check_id": "morgan_011_02",
  "action_id": "morgan_011_send_email"
}
4. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Gives written notice making the Kestrel arrangement end October 14, 2026, or unambiguously 30 days after this September 14 notice. A different effective date or immediate termination fails. No recital of the old dispute is required.",
  "tool": "send_email",
  "check_id": "morgan_011_03",
  "action_id": "morgan_011_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_011_00",
        "action_id": "morgan_011_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "kara@kestrel-test.com",
        "tool": "send_email",
        "check_id": "morgan_011_01",
        "action_id": "morgan_011_send_email"
      },
      {
        "type": "field_list_includes",
        "path": "result.cc",
        "values": [
          "sarah@atlas-test.com"
        ],
        "tool": "send_email",
        "check_id": "morgan_011_02",
        "action_id": "morgan_011_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Gives written notice making the Kestrel arrangement end October 14, 2026, or unambiguously 30 days after this September 14 notice. A different effective date or immediate termination fails. No recital of the old dispute is required.",
        "tool": "send_email",
        "check_id": "morgan_011_03",
        "action_id": "morgan_011_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/011.yaml

SHA-256: 8d539b5577296699e7a6eedc971eff28a6572383ebb53a24ab57a902f2f9a085