Test 089 for Morgan Chen

Request

Email account-team@scaffold.dev what the September 2024 Mercury enterprise release actually shipped and whether it completed enterprise readiness.

Required memory

Fact 199

The shipped slice is narrowly scoped to SAML SSO for scoped enterprise accounts, admin-side audit-history events for invite, source, and admin changes, and clearer separation between admins and billing owners; it does not include advanced admin controls or the standalone procurement/security packet.

Source evidence (1)

001388Sep 13, 2024 / 15:28 UTC-07:00

Mercury enterprise slice made the 3pm cutoff. Leo cleared the evidence review before any release language went out, and Jake/Marcus kept the deploy on the normal release path. The #eng-releases note is intentionally narrow: SAML SSO for scoped enterprise accounts, admin-side audit-history events for invite/source/admin changes, and clearer admin vs billing-owner separation. Sarah has Evergreen-safe wording on those shipped pieces. Devon is keeping the procurement/security packet separate because advanced admin controls and the standalone packet are still not done. Post-release framing: Devon owns commercial/procurement language, Sarah coordinates the customer thread, and we still should not make custom Evergreen promises or say Mercury is fully enterprise-ready.

Message 001388 in history

Fact 200

Post-release communications must not make custom Evergreen promises or claim that Mercury is fully enterprise-ready.

Source evidence (1)

001388Sep 13, 2024 / 15:28 UTC-07:00

Mercury enterprise slice made the 3pm cutoff. Leo cleared the evidence review before any release language went out, and Jake/Marcus kept the deploy on the normal release path. The #eng-releases note is intentionally narrow: SAML SSO for scoped enterprise accounts, admin-side audit-history events for invite/source/admin changes, and clearer admin vs billing-owner separation. Sarah has Evergreen-safe wording on those shipped pieces. Devon is keeping the procurement/security packet separate because advanced admin controls and the standalone packet are still not done. Post-release framing: Devon owns commercial/procurement language, Sarah coordinates the customer thread, and we still should not make custom Evergreen promises or say Mercury is fully enterprise-ready.

Message 001388 in history

Expected tool calls

send_email

Grading

1. field_equals / send_email

{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_089_00",
  "action_id": "morgan_089_send_email"
}

2. field_equals / send_email

{
  "type": "field_equals",
  "path": "result.to",
  "value": "account-team@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_089_01",
  "action_id": "morgan_089_send_email"
}

3. field_llm_judge / send_email

{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Identifies scoped-enterprise SAML SSO, admin audit events for invite/source/admin changes, and clearer admin versus billing-owner separation as shipped. Does not call enterprise readiness complete: advanced admin controls and standalone procurement/security packaging were still unfinished at that release.",
  "tool": "send_email",
  "check_id": "morgan_089_02",
  "action_id": "morgan_089_send_email"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_089_00",
        "action_id": "morgan_089_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "account-team@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_089_01",
        "action_id": "morgan_089_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Identifies scoped-enterprise SAML SSO, admin audit events for invite/source/admin changes, and clearer admin versus billing-owner separation as shipped. Does not call enterprise readiness complete: advanced admin controls and standalone procurement/security packaging were still unfinished at that release.",
        "tool": "send_email",
        "check_id": "morgan_089_02",
        "action_id": "morgan_089_send_email"
      }
    ]
  }
}