DolphinBench
DiscordGitHub

Test 169

Sep 14, 2026 / 1 fact

YAML

Request

Email support@scaffold.dev with the decision about what to do with the old API v2 auth-refresh retry notification draft and whether customers need to act.

Required memory

Fact 142

Morgan Chen requested a concise release note in #eng-releases only, not #eng-all, stating that the API v2 auth refresh retry fix is live and requires no customer action.

Source evidence (1)

000757Oct 3, 2023 / 14:38 UTC-07:00

Marcus says the JWT retry patch is green and is asking how broadly to announce it. Please post a concise release note to #eng-releases only, not #eng-all: the API v2 auth refresh retry fix is live and no customer action is needed. Also DM Marcus in our internal team chat telling him not to post it in the broad engineering channel.

Message 000757 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_169_00",
  "action_id": "morgan_169_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "path": "result.to",
  "value": "support@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_169_01",
  "action_id": "morgan_169_send_email"
}
3. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "States the auth-refresh retry fix is live and no customer action is needed. Does not describe an unfinished rollout or ask customers to migrate.",
  "tool": "send_email",
  "check_id": "morgan_169_02",
  "action_id": "morgan_169_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_169_00",
        "action_id": "morgan_169_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "support@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_169_01",
        "action_id": "morgan_169_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "States the auth-refresh retry fix is live and no customer action is needed. Does not describe an unfinished rollout or ask customers to migrate.",
        "tool": "send_email",
        "check_id": "morgan_169_02",
        "action_id": "morgan_169_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/169.yaml

SHA-256: 68c4aaa7c982504101f963165c66e8cfdcc197c083df055f50c78264076c993d