Test 150

Sep 14, 2026 / 3 facts

Request

Email support@scaffold.dev what Acme could do about its API v2 export timeout at the May 21-22, 2025 checkpoint, before the later patch and closeout.

Required memory

Fact 229

On May 21, 2025, Leo reported that a smaller Acme API v2 replay scoped to account-health and recent renewal-risk rows completed, with no evidence of corrupt exported data.

Source evidence (1)

001655May 21, 2025 / 10:18 UTC-07:00

Leo: request path is API v2 export/full account-health plus renewal-risk bundle. Timeout occurs on the large export path. This does not look like GraphQL auth/JWT, webhook verification, or connector verifier behavior. Smaller replay scoped to account-health and recent renewal-risk rows completed. No evidence of corrupt exported data.

Message 001655 in history

Fact 230

On May 22, 2025, Morgan Chen requested that Sarah be told to recommend Acme proceed with the narrower replay for the renewal-risk review, avoid retrying the full export that day, describe the full export-path timeout as a bounded Atlas/API v2 follow-up, and keep the wording date-free and support-owned.

Source evidence (1)

001661May 22, 2025 / 08:42 UTC-07:00

Tell her: recommend Acme proceed with the narrower replay for the renewal-risk review. Don't have them keep retrying the full export today; the full export-path timeout is our bounded Atlas/API v2 follow-up. Keep it date-free and support-owned.

Message 001661 in history

Fact 231

On May 22, 2025, Leo reported no customer-visible data loss from Acme’s export issue.

Source evidence (1)

001662May 22, 2025 / 08:42 UTC-07:00

Update from Sarah and Leo: Acme asked whether they should rerun the full export before the renewal-risk review; Greg would prefer all rows but can proceed with the narrower replay. Leo says the full export path still times out, the narrower replay completes, there's no customer-visible data loss, and the next technical work is export pagination/timeout guard. Jake says the product follow-up remains Atlas/API v2 sequencing.

Message 001662 in history

Expected tool calls

send_email

Grading

1. field_equals / send_email

{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_150_00",
  "action_id": "morgan_150_send_email"
}

2. field_equals / send_email

{
  "type": "field_equals",
  "path": "result.to",
  "value": "support@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_150_01",
  "action_id": "morgan_150_send_email"
}

3. field_llm_judge / send_email

{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Recommends proceeding with the successful narrower replay rather than repeatedly retrying the full export, which still timed out. Says no corruption/customer-visible data loss was found, and keeps the open full-export fix as bounded Atlas/API v2 support/product follow-up. Does not backdate the later final patch or closure.",
  "tool": "send_email",
  "check_id": "morgan_150_02",
  "action_id": "morgan_150_send_email"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_150_00",
        "action_id": "morgan_150_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "support@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_150_01",
        "action_id": "morgan_150_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Recommends proceeding with the successful narrower replay rather than repeatedly retrying the full export, which still timed out. Says no corruption/customer-visible data loss was found, and keeps the open full-export fix as bounded Atlas/API v2 support/product follow-up. Does not backdate the later final patch or closure.",
        "tool": "send_email",
        "check_id": "morgan_150_02",
        "action_id": "morgan_150_send_email"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/morgan/150.yaml

SHA-256: 143851715670935a494eab110ca3d1343bc0e310ba34733e93d80d8d1edcc86b