DolphinBench

Test 100

Sep 14, 2026 / 4 facts

YAML

Request

Create a document titled Sep 14 support-triage summary using my usual summary format from the notes below. Notes: Acme asked where the export documentation explains resuming a request after a connection drops. Their follow-up asks whether the example covers a long date range. These are documentation questions, not a report that an export failed. Evergreen asked whether the usage wording in its help text refers to active developers or every invited teammate, and requested an example for someone invited but not active that month. Mercury has three source-specific tickets: one reports an invitation email that did not arrive, another asks about an unexpected role label, and the third concerns a teammate list that did not refresh after a change. They arrived separately; there is no evidence here of one shared cause. Atlas has a JWT clock-skew question. The customer wants to know which timestamps to include in the support thread. The on-call handoff reports no customer-impacting incident. A second handoff entry repeats the Evergreen wording question and should not be mistaken for another customer or incident.

Required memory

Fact 39

Morgan Chen wants summaries limited to five bullets.

Source evidence (1)

000305Apr 8, 2023 / 11:00 UTC-07:00

for summaries i want the same thing every time or i stop reading. 5 bullets max.

Message 000305 in history

Fact 40

Morgan Chen requested that each summary bullet be short, with roughly 15 words as a hard cap.

Source evidence (1)

000306Apr 8, 2023 / 11:00 UTC-07:00

and keep each bullet short. 15 words-ish, hard cap.

Message 000306 in history

Fact 41

Morgan Chen requested that summaries have no introductory or concluding line, including no “here’s a quick summary” preface.

Source evidence (1)

000307Apr 8, 2023 / 11:00 UTC-07:00

no intro line. no outro. don't do the 'here's a quick summary' thing.

Message 000307 in history

Fact 42

Morgan Chen wants summaries to contain only bullets and no emoji.

Source evidence (1)

000308Apr 8, 2023 / 11:00 UTC-07:00

also no emoji lol. just the bullets.

Message 000308 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_100_00",
  "action_id": "morgan_100_create_doc"
}
2. field_markdown_bullets / create_doc
{
  "type": "field_markdown_bullets",
  "path": "result.document.body",
  "value": {
    "max_items": 5,
    "max_words": 15,
    "only_bullets": true
  },
  "tool": "create_doc",
  "check_id": "morgan_100_01",
  "action_id": "morgan_100_create_doc"
}
3. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Contains substantive summary points, not introductory or concluding wrappers disguised as bullets. Contains no emoji. Do not demand five bullets when fewer convey the supplied notes.",
  "tool": "create_doc",
  "check_id": "morgan_100_02",
  "action_id": "morgan_100_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_100_00",
        "action_id": "morgan_100_create_doc"
      },
      {
        "type": "field_markdown_bullets",
        "path": "result.document.body",
        "value": {
          "max_items": 5,
          "max_words": 15,
          "only_bullets": true
        },
        "tool": "create_doc",
        "check_id": "morgan_100_01",
        "action_id": "morgan_100_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Contains substantive summary points, not introductory or concluding wrappers disguised as bullets. Contains no emoji. Do not demand five bullets when fewer convey the supplied notes.",
        "tool": "create_doc",
        "check_id": "morgan_100_02",
        "action_id": "morgan_100_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/100.yaml

SHA-256: 7cb97618f4ea29dc27a77b88692fb3102624b14d39d9d2b4351097a98511e1c6