DolphinBench

Test 029

Sep 14, 2026 / 1 fact

YAML

Request

Create a customer-support macro for canceling OAuth before a source connects in Mercury. Explain what happened and how the customer can retry.

Required memory

Fact 271

QA confirmed by January 14, 2026 that Mercury’s OAuth-cancel path tells users authorization was canceled before Scaffold connected the source, says no data was synced, and provides a clear `Try again` path when connector context can be preserved; providers that do not preserve connector context fall back to the generic source picker on the first pass.

Source evidence (1)

002566Jan 14, 2026 / 12:52 UTC-08:00

Closing the loop: Priya and Jake shipped the January Mercury OAuth-cancel activation fix. Production deploy is done, and QA confirmed the cancel path now tells the user authorization was canceled before Scaffold connected the source, says no data was synced, and gives a clear `Try again` path when connector context can be preserved. For providers that don’t preserve connector context, the first pass falls back to the generic source picker. QA also confirmed the successful OAuth callback path is untouched — workspace creation, source connection, and first live sync still pass. No admin-role work, granular-role scope, SAML change, or growth-proof instrumentation shipped.

Message 002566 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_029_00",
  "action_id": "morgan_029_create_doc"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Says no data was synced. Explains retrying with the preserved connector context when available, otherwise using the generic source picker. Does not claim a source connected or data synced after authorization was canceled.",
  "tool": "create_doc",
  "check_id": "morgan_029_01",
  "action_id": "morgan_029_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_029_00",
        "action_id": "morgan_029_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Says no data was synced. Explains retrying with the preserved connector context when available, otherwise using the generic source picker. Does not claim a source connected or data synced after authorization was canceled.",
        "tool": "create_doc",
        "check_id": "morgan_029_01",
        "action_id": "morgan_029_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/029.yaml

SHA-256: d9d8da87c5808dcf8ac16d2f1a376e2f5a34a15126750d6385a581c0892ff40b