DolphinBench
DiscordGitHub

Test 001

Sep 14, 2026 / 1 fact

YAML

Request

I'm out early. Please order one small latte for pickup.

Required memory

Fact 2

Morgan Chen gets Blue Bottle when out early enough.

Source evidence (1)

000001Jan 9, 2023 / 08:00 UTC-08:00

ok so. morgan chen. Scaffold is the company Devon and I started -- developer data plumbing, basically. customers create a workspace, connect their data sources, keep them synced, then use the data through our APIs and connectors. mission apartment, kibo shedding on everything. idk, useful to say it plainly once. 2 years in, still tabs not spaces, still doing blue bottle if i'm out early enough. jamie's on evening shifts again this month so the apartment's been weirdly quiet.

Message 000001 in history

Expected tool calls

  • place_order

Grading

1. field_equals / place_order
{
  "type": "field_equals",
  "tool": "place_order",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_001_00",
  "action_id": "morgan_001_place_order"
}
2. field_llm_judge / place_order
{
  "type": "field_llm_judge",
  "path": "result.order.restaurant",
  "criterion": "The order is from Blue Bottle; Blue Bottle Coffee is an equivalent business name. A generic cafe or another business fails.",
  "tool": "place_order",
  "check_id": "morgan_001_01",
  "action_id": "morgan_001_place_order"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "place_order",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_001_00",
        "action_id": "morgan_001_place_order"
      },
      {
        "type": "field_llm_judge",
        "path": "result.order.restaurant",
        "criterion": "The order is from Blue Bottle; Blue Bottle Coffee is an equivalent business name. A generic cafe or another business fails.",
        "tool": "place_order",
        "check_id": "morgan_001_01",
        "action_id": "morgan_001_place_order"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/001.yaml

SHA-256: bb0aa173345a0dc6ea33279bc94f931c8cf9fe87b4d7ce3edf2d423a27f34a3f