DolphinBench

Test 170

Sep 14, 2026 / 2 facts

YAML

Request

On mercury-release-notes#417, correct the historical expired-magic-link patch description to its actual status and scope.

Required memory

Fact 139

In Morgan Chen’s September 21, 2023 message, Marcus reported that the expired-magic-link helper-text patch was green.

Source evidence (1)

000738Sep 21, 2023 / 14:37 UTC-07:00

Marcus says the expired-magic-link helper-text patch is green and is asking whether to announce it broadly. Please post a short release note to #eng-releases only, not #eng-all: the change clarifies retry guidance for expired magic links, does not change the activation definition, and does not expand auth-provider, SSO, admin, or procurement scope.

Message 000738 in history

Fact 140

Morgan Chen requested a short release note about the expired-magic-link helper-text patch be posted only to #eng-releases, not #eng-all, stating that the change clarifies retry guidance for expired magic links, does not change the activation definition, and does not expand auth-provider, SSO, admin, or procurement scope.

Source evidence (1)

000738Sep 21, 2023 / 14:37 UTC-07:00

Marcus says the expired-magic-link helper-text patch is green and is asking whether to announce it broadly. Please post a short release note to #eng-releases only, not #eng-all: the change clarifies retry guidance for expired magic links, does not change the activation definition, and does not expand auth-provider, SSO, admin, or procurement scope.

Message 000738 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_170_00",
  "action_id": "morgan_170_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "mercury-release-notes#417",
  "tool": "post_pr_comment",
  "check_id": "morgan_170_01",
  "action_id": "morgan_170_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Describes a green/cleared patch clarifying expired-link retry guidance only. Does not change activation definition or expand auth-provider, SSO, admin, or procurement scope. No separate sentence for every excluded area is required.",
  "tool": "post_pr_comment",
  "check_id": "morgan_170_02",
  "action_id": "morgan_170_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_170_00",
        "action_id": "morgan_170_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "mercury-release-notes#417",
        "tool": "post_pr_comment",
        "check_id": "morgan_170_01",
        "action_id": "morgan_170_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Describes a green/cleared patch clarifying expired-link retry guidance only. Does not change activation definition or expand auth-provider, SSO, admin, or procurement scope. No separate sentence for every excluded area is required.",
        "tool": "post_pr_comment",
        "check_id": "morgan_170_02",
        "action_id": "morgan_170_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/170.yaml

SHA-256: a43ca76813fe9e76e36607aab985293d8a8c6367ead801d2de19f187fe248ce0