DolphinBench

Test 195

Sep 14, 2026 / 2 facts

YAML

Request

Create a 2023 retreat archive note identifying the organizer and budget owner.

Required memory

Fact 60

Sarah is coordinating the retreat this year.

Source evidence (1)

000372Apr 15, 2023 / 14:00 UTC-07:00

oh and on retreat stuff, loop Sarah in. budget still runs through HR but she's the one actually herding it this year

Message 000372 in history

Fact 137

Morgan Chen requested that HR be kept as the budget owner.

Source evidence (1)

000704Aug 29, 2023 / 09:29 UTC-07:00

HR needs the final retreat catering count lock. From: HR To: Morgan Chen Cc: Sarah Kim <sarah@atlas-test.com> Date: Tue, 29 Aug 2023 09:12 AM Subject: Retreat catering count lock — due today Hi Morgan, We need the final lunch count lock today. Tava Kitchen can hold the order at the following counts if we confirm by 2:00 p.m.: - Total attendees: 33 - Standard meals: 23 - Vegetarian: 5 - Vegan: 3 - Gluten-free: 2 Optional add-on: afternoon snack service is $420 before tax and service if the team wants it added. Please reply on this thread with a yes/no on the lunch counts by 2:00 p.m. HR will keep the budget line on our side. Thanks, HR Please reply on the count-lock email thread and cc Sarah Kim. Confirm the Tava Kitchen lunch counts exactly as listed: total 33, 23 standard, 5 vegetarian, 3 vegan, and 2 gluten-free. Keep HR as the budget owner, and decline the afternoon snack service or any dinner/snack add-ons unless HR provides a separate approved budget path.

Message 000704 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_195_00",
  "action_id": "morgan_195_create_doc"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Identifies Sarah as organizer/contact and HR as budget owner for that retreat. Does not substitute the organizer for budget approval or make it a timeless assignment.",
  "tool": "create_doc",
  "check_id": "morgan_195_01",
  "action_id": "morgan_195_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_195_00",
        "action_id": "morgan_195_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Identifies Sarah as organizer/contact and HR as budget owner for that retreat. Does not substitute the organizer for budget approval or make it a timeless assignment.",
        "tool": "create_doc",
        "check_id": "morgan_195_01",
        "action_id": "morgan_195_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/195.yaml

SHA-256: 25166b1023f39a767992fe4e563ee34813b00fd4273e5a32aeaee8b3acd50030