DolphinBench

Test 128

Sep 14, 2026 / 2 facts

YAML

Request

Check Mercury, Atlas, and Compass, then create today's manager update in my usual status-update format. Mercury implementation note: The teammate list now refreshes when a role-change request completes instead of waiting for the next poll. The occasional pending banner can still remain for about 30 seconds. Atlas handoff: Review the cursor-versus-fresh-poll, webhook, and JWT examples before the next support walkthrough. Compass handoff: Check the archived screenshot captions before the September 18 review.

Required memory

Fact 66

On April 20, 2023, Morgan Chen said the latest status update was too long and requested that future status updates contain only highlights in 3–4 bullets maximum.

Source evidence (1)

000403Apr 20, 2023 / 09:30 UTC-07:00

status update was way too long. just give me the highlights next time, like 3-4 bullets max

Message 000403 in history

Fact 67

Morgan Chen wants detailed explanations reserved for code-related content.

Source evidence (1)

000404Apr 20, 2023 / 09:30 UTC-07:00

details only if it's code stuff. otherwise i can skim the bullets and we're good

Message 000404 in history

Expected tool calls

  • check_project_status
  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_128_00",
  "action_id": "morgan_128_create_doc"
}
2. field_markdown_bullets / create_doc
{
  "type": "field_markdown_bullets",
  "path": "result.document.body",
  "value": {
    "max_items": 4
  },
  "tool": "create_doc",
  "check_id": "morgan_128_01",
  "action_id": "morgan_128_create_doc"
}
3. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Uses short substantive status bullets rather than a detailed non-code narrative. Reports all three projects without contradiction: Mercury's role-change refresh implementation may receive extra technical detail, the occasional pending banner remains, and shipment is still conditional; Atlas's examples need review and its latest recorded case matched the API v2 contract without showing a product problem; Compass's caption review does not complete its still-open September 18 evidence gate. Do not require the Atlas bullet to list every absent problem, such as server errors and data loss. Brief labels are acceptable.",
  "tool": "create_doc",
  "check_id": "morgan_128_02",
  "action_id": "morgan_128_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_128_00",
        "action_id": "morgan_128_create_doc"
      },
      {
        "type": "field_markdown_bullets",
        "path": "result.document.body",
        "value": {
          "max_items": 4
        },
        "tool": "create_doc",
        "check_id": "morgan_128_01",
        "action_id": "morgan_128_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Uses short substantive status bullets rather than a detailed non-code narrative. Reports all three projects without contradiction: Mercury's role-change refresh implementation may receive extra technical detail, the occasional pending banner remains, and shipment is still conditional; Atlas's examples need review and its latest recorded case matched the API v2 contract without showing a product problem; Compass's caption review does not complete its still-open September 18 evidence gate. Do not require the Atlas bullet to list every absent problem, such as server errors and data loss. Brief labels are acceptable.",
        "tool": "create_doc",
        "check_id": "morgan_128_02",
        "action_id": "morgan_128_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/128.yaml

SHA-256: 57d50d05d8a64b22b2d670864d8c51fe7eb5a6157797910dec4fd08047333d90