DolphinBench

Test 108

Sep 14, 2026 / 2 facts

YAML

Request

On scaffold/mercury#3120, correct the claim that the December 14, 2023 hardening deploy shipped SSO and broader admin/billing separation. Say what actually shipped.

Required memory

Fact 160

The December 14, 2023 Mercury hardening deploy put two scoped fixes live: the org-invite admin-setup label changed from “Billing owner” to “Workspace admin,” and the live-sync retry CTA is disabled while a retry is already in flight.

Source evidence (1)

000922Dec 14, 2023 / 10:48 UTC-08:00

The deploy is green and I have Leo/Jake/Rishi’s wording below: Discord DM excerpts — Thu, Dec 14, 2023 [Leo Park → Morgan Chen — 10:18 AM PT] Mercury hardening deploy is green. Checks from the post-deploy pass: - deploy completed cleanly at 10:11 PT - post-deploy smoke on the current admin invite/setup path is green - retry-in-flight guard behaved correctly on the bounded live-sync replay account; CTA disabled while the retry was already running - no duplicate sync job seen in logs during the replay - error rate / alerting flat through the first 20 min - broader MER-1749 state-reconcile work is still out of scope and still not in this deploy Proposed #eng-releases note: "Mercury hardening update is now live. This deploy includes two scoped fixes: (1) clearer admin-setup wording on the org invite step (\"Workspace admin\" replaces the old \"Billing owner\" label), and (2) a narrow guard that disables the live-sync retry CTA while a retry is already in flight. No broader live-sync state-reconcile change, SSO work, audit-history work, or admin-versus-billing-owner model changes are part of this deploy." If you want tighter wording before it goes out, I can hold. [Jake → Morgan Chen — 10:23 AM PT] If green is official, can you send me the final release-note wording? I want the retry line to stay tight and not read like we fixed the whole retry-state story. Also assuming this is #eng-releases only unless you want something else. [Rishi → Morgan Chen — 10:36 AM PT] Support blurb draft below. One sentence may overcall it. "We shipped a small Mercury setup/recovery hardening update. The org-invite step now uses clearer workspace-admin wording, and the retry action during first live sync is disabled while a previous retry is already running, which makes recovery behavior clearer. This should make the current Mercury admin flow feel more enterprise-ready for teams evaluating it." Possible safer last sentence: "This should make the current Mercury admin flow clearer during setup and first-sync recovery." Super-short version if support wants it: "No action required; this is a UI/guardrail cleanup, not a change to auth or admin permissions." Please post the internal release note to #eng-releases now. Keep it to the two current fixes: admin-setup “Billing owner” mislabel changed to “Workspace admin” wording, and the live-sync retry guard while a retry is already in flight. Do not post to #eng-all. Also send Jake and Rishi private current-team-chat notes: support wording should not name Evergreen, and nothing here implies SSO, audit history, admin-versus-billing-owner separation, broader state reconcile, or enterprise readiness shipped.

Message 000922 in history

Fact 161

Morgan Chen requested a private current-team-chat note to Jake stating that Mercury support wording must not name Evergreen or imply that SSO, audit history, admin-versus-billing-owner separation, broader state reconciliation, or enterprise readiness shipped in the December 14, 2023 deploy.

Source evidence (1)

000922Dec 14, 2023 / 10:48 UTC-08:00

The deploy is green and I have Leo/Jake/Rishi’s wording below: Discord DM excerpts — Thu, Dec 14, 2023 [Leo Park → Morgan Chen — 10:18 AM PT] Mercury hardening deploy is green. Checks from the post-deploy pass: - deploy completed cleanly at 10:11 PT - post-deploy smoke on the current admin invite/setup path is green - retry-in-flight guard behaved correctly on the bounded live-sync replay account; CTA disabled while the retry was already running - no duplicate sync job seen in logs during the replay - error rate / alerting flat through the first 20 min - broader MER-1749 state-reconcile work is still out of scope and still not in this deploy Proposed #eng-releases note: "Mercury hardening update is now live. This deploy includes two scoped fixes: (1) clearer admin-setup wording on the org invite step (\"Workspace admin\" replaces the old \"Billing owner\" label), and (2) a narrow guard that disables the live-sync retry CTA while a retry is already in flight. No broader live-sync state-reconcile change, SSO work, audit-history work, or admin-versus-billing-owner model changes are part of this deploy." If you want tighter wording before it goes out, I can hold. [Jake → Morgan Chen — 10:23 AM PT] If green is official, can you send me the final release-note wording? I want the retry line to stay tight and not read like we fixed the whole retry-state story. Also assuming this is #eng-releases only unless you want something else. [Rishi → Morgan Chen — 10:36 AM PT] Support blurb draft below. One sentence may overcall it. "We shipped a small Mercury setup/recovery hardening update. The org-invite step now uses clearer workspace-admin wording, and the retry action during first live sync is disabled while a previous retry is already running, which makes recovery behavior clearer. This should make the current Mercury admin flow feel more enterprise-ready for teams evaluating it." Possible safer last sentence: "This should make the current Mercury admin flow clearer during setup and first-sync recovery." Super-short version if support wants it: "No action required; this is a UI/guardrail cleanup, not a change to auth or admin permissions." Please post the internal release note to #eng-releases now. Keep it to the two current fixes: admin-setup “Billing owner” mislabel changed to “Workspace admin” wording, and the live-sync retry guard while a retry is already in flight. Do not post to #eng-all. Also send Jake and Rishi private current-team-chat notes: support wording should not name Evergreen, and nothing here implies SSO, audit history, admin-versus-billing-owner separation, broader state reconcile, or enterprise readiness shipped.

Message 000922 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_108_00",
  "action_id": "morgan_108_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/mercury#3120",
  "tool": "post_pr_comment",
  "check_id": "morgan_108_01",
  "action_id": "morgan_108_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Limits that deploy to replacing the Billing owner label with Workspace admin on org setup and disabling retry while a retry is in flight. Rejects the claimed SSO/admin-role-model expansion and does not imply broader state reconciliation shipped.",
  "tool": "post_pr_comment",
  "check_id": "morgan_108_02",
  "action_id": "morgan_108_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_108_00",
        "action_id": "morgan_108_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/mercury#3120",
        "tool": "post_pr_comment",
        "check_id": "morgan_108_01",
        "action_id": "morgan_108_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Limits that deploy to replacing the Billing owner label with Workspace admin on org setup and disabling retry while a retry is in flight. Rejects the claimed SSO/admin-role-model expansion and does not imply broader state reconciliation shipped.",
        "tool": "post_pr_comment",
        "check_id": "morgan_108_02",
        "action_id": "morgan_108_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/108.yaml

SHA-256: 3f83ba58407f371a85d36ceeb87d5dd8b1cf092dad881cea45a55e191b62189f