DolphinBench

Test 010

Sep 14, 2026 / 1 fact

YAML

Request

Email finance@scaffold.dev a reminder of my invoice-review rule. State the exact dollar boundary, when an invoice must be flagged, and whether manual review is optional.

Required memory

Fact 36

As of April 7, 2023, finance requires every payment over $5,000 to be flagged and manually reviewed before payment; the rule is non-negotiable.

Source evidence (1)

000300Apr 7, 2023 / 09:00 UTC-07:00

finance one is non-negotiable now: anything over $5,000 gets flagged before we pay it. manual review, every time.

Message 000300 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_010_00",
  "action_id": "morgan_010_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.to",
  "value": "finance@scaffold.dev",
  "check_id": "morgan_010_01",
  "action_id": "morgan_010_send_email"
}
3. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "tool": "send_email",
  "path": "result.body",
  "criterion": "States that every payment over $5,000 must be flagged before payment and manually reviewed every time, and that this rule is not optional.",
  "check_id": "morgan_010_02",
  "action_id": "morgan_010_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_010_00",
        "action_id": "morgan_010_send_email"
      },
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.to",
        "value": "finance@scaffold.dev",
        "check_id": "morgan_010_01",
        "action_id": "morgan_010_send_email"
      },
      {
        "type": "field_llm_judge",
        "tool": "send_email",
        "path": "result.body",
        "criterion": "States that every payment over $5,000 must be flagged before payment and manually reviewed every time, and that this rule is not optional.",
        "check_id": "morgan_010_02",
        "action_id": "morgan_010_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/010.yaml

SHA-256: b337402f9da68484348d8522c53daf5ca1b21414a3059713ed922f447ecafafc