DolphinBench
DiscordGitHub

Test 050

Sep 14, 2026 / 1 fact

YAML

Request

Email sarah@scaffold.dev answer language for Evergreen Finance asking whether its annual account includes a custom admin package or an overage cap.

Required memory

Fact 274

Evergreen Bank’s February 1, 2026 through January 31, 2027 signed renewal includes quarterly current-state/admin reviews only; it excludes a granular sync-operator commitment, custom admin package, overage cap, department carveout, roadmap appendix, delivery date, and advanced-admin renewal obligation.

Source evidence (1)

002614Jan 27, 2026 / 09:36 UTC-08:00

Sarah and Devon got the completed Evergreen signature packet this morning. Please update the Evergreen Bank CRM row now so the commercial status is clear and nobody treats this as a custom package or routes follow-up back to me. It is an annual Mercury Growth renewal for February 1, 2026 through January 31, 2027 on standard terms: $7,500 per month including up to 200 monthly active developers, plus $1,000 per additional 50 monthly active developers. The signed agreement includes quarterly current-state/admin reviews only and does not include a granular sync-operator commitment, custom admin package, overage cap, department carveout, roadmap appendix, delivery date, or advanced-admin renewal obligation.

Message 002614 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_050_00",
  "action_id": "morgan_050_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "path": "result.to",
  "value": "sarah@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_050_01",
  "action_id": "morgan_050_send_email"
}
3. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Clearly says the annual account includes neither a custom admin package nor an overage cap. Does not require repeating the entire annual agreement or its pricing.",
  "tool": "send_email",
  "check_id": "morgan_050_02",
  "action_id": "morgan_050_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_050_00",
        "action_id": "morgan_050_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "sarah@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_050_01",
        "action_id": "morgan_050_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Clearly says the annual account includes neither a custom admin package nor an overage cap. Does not require repeating the entire annual agreement or its pricing.",
        "tool": "send_email",
        "check_id": "morgan_050_02",
        "action_id": "morgan_050_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/050.yaml

SHA-256: 3c3c2b1b4071a543165a981b0ee7073ecb3db10e2fbe63d5e4537208f39257d1