DolphinBench

Test 133

Sep 14, 2026 / 4 facts

YAML

Request

Email Kestrel Marketing my publication decision on its Mercury page draft, which mixes shipped capabilities with named-customer proof, financing news, and a broad enterprise-readiness headline.

Required memory

Fact 34

Morgan Chen requested that Sarah Kim (sarah@atlas-test.com) be CC'd on outside email threads from the start so she can see the full thread.

Source evidence (1)

000292Apr 6, 2023 / 10:00 UTC-07:00

ok so for outside email, just cc Sarah Kim on it. sarah@atlas-test.com. easier if she sees the thread from the start

Message 000292 in history

Fact 72

Kestrel Marketing’s contact is Kara at kara@kestrel-test.com.

Source evidence (1)

000428Apr 23, 2023 / 16:00 UTC-07:00

third contractor this month with the exact same mess. Kestrel Marketing now too. every deliverable slipping, just took longer to show itself than Acme and Blueline. tired of being right about this stuff. Kara at Kestrel is kara@kestrel-test.com. and for the Blueline wind-down ops contact is still the one to use.

Message 000428 in history

Fact 204

Kestrel’s Mercury maintenance pass went live after Sarah’s review, with the public page claiming only the shipped scoped SAML SSO, admin-audit visibility, and clearer admin-role separation, all in generic product language.

Source evidence (1)

001423Oct 25, 2024 / 15:47 UTC-07:00

Kestrel’s Mercury maintenance pass is live after Sarah’s review. Kara confirmed it stayed as a maintenance update: the public page only claims the shipped scoped SAML SSO, admin-audit visibility, and clearer admin-role separation, all in generic product language. Anything Evergreen-specific, customer-proof-y, funding-related, or broader enterprise-readiness is still on the separate Scaffold review path.

Message 001423 in history

Fact 205

Kara confirmed that Evergreen-specific content, customer-proof material, funding-related language, and broader enterprise-readiness claims are excluded from the live Mercury maintenance update and are on the separate Scaffold review path.

Source evidence (1)

001423Oct 25, 2024 / 15:47 UTC-07:00

Kestrel’s Mercury maintenance pass is live after Sarah’s review. Kara confirmed it stayed as a maintenance update: the public page only claims the shipped scoped SAML SSO, admin-audit visibility, and clearer admin-role separation, all in generic product language. Anything Evergreen-specific, customer-proof-y, funding-related, or broader enterprise-readiness is still on the separate Scaffold review path.

Message 001423 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_133_00",
  "action_id": "morgan_133_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "path": "result.to",
  "value": "kara@kestrel-test.com",
  "tool": "send_email",
  "check_id": "morgan_133_01",
  "action_id": "morgan_133_send_email"
}
3. field_list_includes / send_email
{
  "type": "field_list_includes",
  "path": "result.cc",
  "values": [
    "sarah@atlas-test.com"
  ],
  "tool": "send_email",
  "check_id": "morgan_133_02",
  "action_id": "morgan_133_send_email"
}
4. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Clears only generic product language for shipped scoped SAML SSO, admin-audit visibility, and clearer admin-role separation. Holds customer-identifying proof, financing claims, and broad enterprise-readiness claims for separate internal Scaffold review, not vendor publication.",
  "tool": "send_email",
  "check_id": "morgan_133_03",
  "action_id": "morgan_133_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_133_00",
        "action_id": "morgan_133_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "kara@kestrel-test.com",
        "tool": "send_email",
        "check_id": "morgan_133_01",
        "action_id": "morgan_133_send_email"
      },
      {
        "type": "field_list_includes",
        "path": "result.cc",
        "values": [
          "sarah@atlas-test.com"
        ],
        "tool": "send_email",
        "check_id": "morgan_133_02",
        "action_id": "morgan_133_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Clears only generic product language for shipped scoped SAML SSO, admin-audit visibility, and clearer admin-role separation. Holds customer-identifying proof, financing claims, and broad enterprise-readiness claims for separate internal Scaffold review, not vendor publication.",
        "tool": "send_email",
        "check_id": "morgan_133_03",
        "action_id": "morgan_133_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/133.yaml

SHA-256: 8755bc962c90344d6c23570cd561e8da1c3b90c688dde0bd8da716f8a97a8b90