Test 125 for Morgan Chen

Request

Email support@scaffold.dev our decision on closing Evergreen's April 2025 SAML rotation ticket and the instructions for routing new SAML issues.

Required memory

Fact 218

The workspace-admin support path is the first-line support route.

Source evidence (1)

001547Apr 1, 2025 / 11:47 UTC-07:00

Prep hold just wrapped. Outcome is clean and I want it written down before this gets mixed up with broader admin/procurement stuff. Notes from Sarah, Leo, Devon, and me: - Sarah confirmed the current Evergreen account thread points to the annual Mercury Growth account and the shipped SAML login path. - Leo confirmed the metadata and cert-upload runbook uses the current shipped SAML path. - Apr 3 test plan: update cert, run IdP login for an existing admin and a workspace-admin user, confirm visible invite/source/admin audit lookup still works, and watch auth logs for 401 or metadata errors for 30 minutes. - Support route: workspace-admin support path stays first line; Leo owns SAML/platform issues; Sarah owns the customer thread; Devon is available only for commercial/procurement context. - Explicit exclusions: no advanced-admin policy, granular role editor, procurement/security add-on, or roadmap commitment. Create an internal prep-notes doc capturing the Apr 1 outcome, Apr 3 verification steps, ownership, and the customer-safe boundary.

Message 001547 in history

Fact 219

Leo owns SAML and platform issues.

Source evidence (1)

001547Apr 1, 2025 / 11:47 UTC-07:00

Prep hold just wrapped. Outcome is clean and I want it written down before this gets mixed up with broader admin/procurement stuff. Notes from Sarah, Leo, Devon, and me: - Sarah confirmed the current Evergreen account thread points to the annual Mercury Growth account and the shipped SAML login path. - Leo confirmed the metadata and cert-upload runbook uses the current shipped SAML path. - Apr 3 test plan: update cert, run IdP login for an existing admin and a workspace-admin user, confirm visible invite/source/admin audit lookup still works, and watch auth logs for 401 or metadata errors for 30 minutes. - Support route: workspace-admin support path stays first line; Leo owns SAML/platform issues; Sarah owns the customer thread; Devon is available only for commercial/procurement context. - Explicit exclusions: no advanced-admin policy, granular role editor, procurement/security add-on, or roadmap commitment. Create an internal prep-notes doc capturing the Apr 1 outcome, Apr 3 verification steps, ownership, and the customer-safe boundary.

Message 001547 in history

Fact 220

Sarah and Leo reported in the April 3, 2025 Evergreen Bank closeout that the SAML certificate rotation was completed.

Source evidence (1)

001552Apr 3, 2025 / 13:36 UTC-07:00

Evergreen is done. Closeout from Sarah and Leo: - Evergreen completed the certificate rotation. - Existing-admin SAML login and workspace-admin SAML login both passed after rotation. - Leo saw no elevated 401s or metadata errors during the 30-minute watch. - The annual Mercury Growth account kept working on the shipped SAML path. - Sarah sent the customer closeout in the existing account thread. - Devon stayed out of customer/procurement follow-up because no commercial/procurement question was raised. Update the Evergreen CRM notes with the completed rotation outcome and close the rotation thread.

Message 001552 in history

Fact 221

Sarah and Leo reported that Evergreen Bank’s existing-admin and workspace-admin SAML logins both passed after the certificate rotation.

Source evidence (1)

001552Apr 3, 2025 / 13:36 UTC-07:00

Evergreen is done. Closeout from Sarah and Leo: - Evergreen completed the certificate rotation. - Existing-admin SAML login and workspace-admin SAML login both passed after rotation. - Leo saw no elevated 401s or metadata errors during the 30-minute watch. - The annual Mercury Growth account kept working on the shipped SAML path. - Sarah sent the customer closeout in the existing account thread. - Devon stayed out of customer/procurement follow-up because no commercial/procurement question was raised. Update the Evergreen CRM notes with the completed rotation outcome and close the rotation thread.

Message 001552 in history

Expected tool calls

send_email

Grading

1. field_equals / send_email

{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_125_00",
  "action_id": "morgan_125_send_email"
}

2. field_equals / send_email

{
  "type": "field_equals",
  "path": "result.to",
  "value": "support@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_125_01",
  "action_id": "morgan_125_send_email"
}

3. field_llm_judge / send_email

{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Closes the old rotation ticket because rotation completed and the annual account worked on the shipped SAML path. Uses workspace-admin support first, with Leo for SAML/platform issues. No unrelated commercial-owner recital is required.",
  "tool": "send_email",
  "check_id": "morgan_125_02",
  "action_id": "morgan_125_send_email"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_125_00",
        "action_id": "morgan_125_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "support@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_125_01",
        "action_id": "morgan_125_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Closes the old rotation ticket because rotation completed and the annual account worked on the shipped SAML path. Uses workspace-admin support first, with Leo for SAML/platform issues. No unrelated commercial-owner recital is required.",
        "tool": "send_email",
        "check_id": "morgan_125_02",
        "action_id": "morgan_125_send_email"
      }
    ]
  }
}