DolphinBench

Test 187

Sep 14, 2026 / 1 fact

YAML

Request

Write slide 1 of Auth_Rewrite_Archive as the historical October 2023 cleanup plan for the staging toggles, rollback flag, and release announcement.

Required memory

Fact 148

Morgan requested a DM to Marcus instructing him to remove the two staging-only Clerk toggles after confirming there are no production references, leave the rollback flag in place until Friday, 2023-11-03, and skip a release note because customer behavior did not change.

Source evidence (1)

000799Oct 25, 2023 / 15:08 UTC-07:00

Marcus says the auth rewrite is still quiet after rollout, but cleanup has two staging-only Clerk toggles and one rollback flag left. Please DM Marcus: remove the staging-only toggles after confirming there are no production references, leave the rollback flag in place until Friday, 2023-11-03, and skip a release note because customer behavior did not change.

Message 000799 in history

Expected tool calls

  • update_deck_slide

Grading

1. field_equals / update_deck_slide
{
  "type": "field_equals",
  "tool": "update_deck_slide",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_187_00",
  "action_id": "morgan_187_update_deck_slide"
}
2. field_equals / update_deck_slide
{
  "type": "field_equals",
  "path": "result.deck",
  "value": "Auth_Rewrite_Archive",
  "tool": "update_deck_slide",
  "check_id": "morgan_187_01",
  "action_id": "morgan_187_update_deck_slide"
}
3. field_eq_number / update_deck_slide
{
  "type": "field_eq_number",
  "path": "result.slide.number",
  "value": 1,
  "tool": "update_deck_slide",
  "check_id": "morgan_187_02",
  "action_id": "morgan_187_update_deck_slide"
}
4. field_llm_judge / update_deck_slide
{
  "type": "field_llm_judge",
  "path": "result.slide.content",
  "criterion": "Requires confirmation of no production references before removing both staging-only Clerk toggles, retains the rollback flag until November 3, 2023, and skips a release note because customer behavior did not change. Describes the historical plan, not unverified completed removal. Naming Marcus is optional; omitting the production-reference prerequisite is not.",
  "tool": "update_deck_slide",
  "check_id": "morgan_187_03",
  "action_id": "morgan_187_update_deck_slide"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "update_deck_slide",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_187_00",
        "action_id": "morgan_187_update_deck_slide"
      },
      {
        "type": "field_equals",
        "path": "result.deck",
        "value": "Auth_Rewrite_Archive",
        "tool": "update_deck_slide",
        "check_id": "morgan_187_01",
        "action_id": "morgan_187_update_deck_slide"
      },
      {
        "type": "field_eq_number",
        "path": "result.slide.number",
        "value": 1,
        "tool": "update_deck_slide",
        "check_id": "morgan_187_02",
        "action_id": "morgan_187_update_deck_slide"
      },
      {
        "type": "field_llm_judge",
        "path": "result.slide.content",
        "criterion": "Requires confirmation of no production references before removing both staging-only Clerk toggles, retains the rollback flag until November 3, 2023, and skips a release note because customer behavior did not change. Describes the historical plan, not unverified completed removal. Naming Marcus is optional; omitting the production-reference prerequisite is not.",
        "tool": "update_deck_slide",
        "check_id": "morgan_187_03",
        "action_id": "morgan_187_update_deck_slide"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/187.yaml

SHA-256: 49d33ec700628429cef4ab6be4c5fec74064307b965503952aac73e344944231