Test 134

Sep 14, 2026 / 3 facts

Request

Update only Evergreen Bank's CRM trigger_condition with how ordinary product requests should be handled and who must review custom scope, feature-date promises, or priority-account exceptions before customer follow-up. Leave every other field unchanged.

Required memory

Fact 198

Under the August 23, 2024 boundary reset, asks resembling custom scope, feature-date promises, or priority-account exceptions must go through Morgan Chen or Devon Hayes and the relevant product owner before anyone uses them in customer follow-up.

Source evidence (1)

001366Aug 23, 2024 / 16:26 UTC-07:00

Nadia’s enterprise prep is getting useful — the Evergreen-style account reads are much sharper — but we had to reset the boundary today. Her draft mixed account patterns, source notes, customer follow-ups, and “roadmap implications,” and a few priority-account asks were written too much like commitments: SSO timing, audit-history completeness, procurement packet timing, admin-role exceptions. Devon and I pulled it back with Jake in the room. Nadia should own pattern capture, account prep, and a clearer conversation map. She does not own bespoke roadmap commitments. If an ask starts to look like custom scope, a feature-date promise, or a priority-account exception, it has to go through me or Devon and the relevant product owner before anyone uses it in customer follow-up.

Message 001366 in history

Fact 217

Morgan Chen requested setting the Evergreen Bank CRM notes exactly to the supplied annual-conversion text covering the February 1, 2025 through January 31, 2026 term, standard Growth pricing, January closeout validations, Sarah Kim’s customer-thread ownership, Devon Hayes’s commercial/procurement framing, Nadia Singh’s customer-growth pattern capture, and product-request routing through the standard Mercury roadmap.

Source evidence (1)

001494Jan 31, 2025 / 09:58 UTC-08:00

Evergreen closed. Please update the existing Evergreen Bank CRM row now so it is not stuck in paid-pilot status. Close email / Devon commercial summary: Evergreen Bank converts from the paid Mercury Growth pilot to an annual Mercury Growth agreement. The annual term is February 1, 2025 through January 31, 2026. The commercial structure is Scaffold’s standard Growth terms. The price is $7,500 per month including up to 200 monthly active developers. Overage is $1,000 per additional 50 monthly active developers. The January closeout validated that SAML login works for the annual account. The January closeout validated that visible invite/source/admin audit lookup is sufficient for the annual deal. Advanced admin controls are not included in the annual baseline. Broader procurement/security add-ons are separately quoted only after the relevant work exists. Sarah Kim remains Evergreen customer-thread owner. Devon Hayes owns commercial/procurement framing. Nadia Singh captures repeatable customer-growth patterns. Product requests continue through the standard Mercury roadmap. Set status exactly to: annual_growth_customer Set tags exactly to: mercury, growth, annual-customer, evergreen Set trigger condition exactly to: Annual Growth agreement runs February 1, 2025 through January 31, 2026; advanced admin controls and broader procurement/security add-ons require separate quoting after relevant work exists. Set notes exactly to: Converted to annual Mercury Growth customer effective 2025-02-01 through 2026-01-31 on standard Growth terms: $7,500/month including up to 200 monthly active developers, plus $1,000 per additional 50 monthly active developers. January closeout validated SAML login and visible invite/source/admin audit lookup. Sarah Kim remains customer-thread owner; Devon Hayes owns commercial/procurement framing; Nadia Singh captures repeatable customer-growth patterns; product requests continue through the standard Mercury roadmap.

Message 001494 in history

Fact 336

The user is Morgan Chen.

Source evidence (1)

000001Jan 9, 2023 / 08:00 UTC-08:00

ok so. morgan chen. Scaffold is the company Devon and I started -- developer data plumbing, basically. customers create a workspace, connect their data sources, keep them synced, then use the data through our APIs and connectors. mission apartment, kibo shedding on everything. idk, useful to say it plainly once. 2 years in, still tabs not spaces, still doing blue bottle if i'm out early enough. jamie's on evening shifts again this month so the apartment's been weirdly quiet.

Message 000001 in history

Expected tool calls

update_crm_row

Grading

1. field_equals / update_crm_row

{
  "type": "field_equals",
  "tool": "update_crm_row",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_134_00",
  "action_id": "morgan_134_update_crm_row"
}

2. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.id",
  "value": "crm_1785189091939",
  "tool": "update_crm_row",
  "check_id": "morgan_134_01",
  "action_id": "morgan_134_update_crm_row"
}

3. field_llm_judge / update_crm_row

{
  "type": "field_llm_judge",
  "path": "result.row.trigger_condition",
  "criterion": "Routes ordinary requests through the standard Mercury roadmap. Requires Morgan or Devon AND the relevant product owner to review custom scope, feature dates, or priority-account exceptions before customer follow-up. Neither reviewer component is optional.",
  "tool": "update_crm_row",
  "check_id": "morgan_134_02",
  "action_id": "morgan_134_update_crm_row"
}

4. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.name",
  "value": "Evergreen Bank",
  "tool": "update_crm_row",
  "check_id": "morgan_134_03",
  "action_id": "morgan_134_update_crm_row"
}

5. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.status",
  "value": "q3_2026_admin_review_closed_current_scope",
  "tool": "update_crm_row",
  "check_id": "morgan_134_04",
  "action_id": "morgan_134_update_crm_row"
}

6. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.next_touch_date",
  "value": "normal Q2 account check-in only",
  "tool": "update_crm_row",
  "check_id": "morgan_134_05",
  "action_id": "morgan_134_update_crm_row"
}

7. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.notes",
  "value": "November 2025 usage closed at 223 monthly active developers; treasury and risk operations are active alongside the original department; Evergreen is processing the standard first Growth overage band of $1,000 under existing terms; no seat carveout, custom department package, or custom commercial exception.\n\nJan 27, 2026: annual Mercury Growth renewal signed for Feb 1, 2026 through Jan 31, 2027 on standard terms: $7,500 per month including up to 200 monthly active developers, plus $1,000 per additional 50 monthly active developers. Quarterly current-state/admin reviews are included for then-current admin controls only. The signed renewal does not include a granular sync-operator commitment, custom admin package, overage cap, department carveout, roadmap appendix, delivery date, or advanced-admin renewal obligation. Treat this as a standard Growth renewal, not a custom package or founder-owned follow-up.\n\nApr 16, 2026: first included 2026 current-state/admin review closed inside standard Growth mechanics. Covered shipped SAML login, visible invite/source/admin audit lookup, workspace-admin support, and March usage under standard Growth. No custom admin policy, overage cap, granular-role date or package, procurement-security item, add-on, or department carveout commitment. Sync-operator interest remains later advanced-admin research only, with no committed date. Sarah Kim owns the customer thread and Devon Hayes covers commercial/procurement context.",
  "tool": "update_crm_row",
  "check_id": "morgan_134_06",
  "action_id": "morgan_134_update_crm_row"
}

8. field_equals / update_crm_row

{
  "type": "field_equals",
  "path": "result.row.tags",
  "value": [
    "mercury",
    "growth",
    "annual-customer",
    "evergreen",
    "current-state-admin-review-closed",
    "granular-sync-operator-logged-input",
    "no-commercial-exception"
  ],
  "tool": "update_crm_row",
  "check_id": "morgan_134_07",
  "action_id": "morgan_134_update_crm_row"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "update_crm_row",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_134_00",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.id",
        "value": "crm_1785189091939",
        "tool": "update_crm_row",
        "check_id": "morgan_134_01",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_llm_judge",
        "path": "result.row.trigger_condition",
        "criterion": "Routes ordinary requests through the standard Mercury roadmap. Requires Morgan or Devon AND the relevant product owner to review custom scope, feature dates, or priority-account exceptions before customer follow-up. Neither reviewer component is optional.",
        "tool": "update_crm_row",
        "check_id": "morgan_134_02",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.name",
        "value": "Evergreen Bank",
        "tool": "update_crm_row",
        "check_id": "morgan_134_03",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.status",
        "value": "q3_2026_admin_review_closed_current_scope",
        "tool": "update_crm_row",
        "check_id": "morgan_134_04",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.next_touch_date",
        "value": "normal Q2 account check-in only",
        "tool": "update_crm_row",
        "check_id": "morgan_134_05",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.notes",
        "value": "November 2025 usage closed at 223 monthly active developers; treasury and risk operations are active alongside the original department; Evergreen is processing the standard first Growth overage band of $1,000 under existing terms; no seat carveout, custom department package, or custom commercial exception.\n\nJan 27, 2026: annual Mercury Growth renewal signed for Feb 1, 2026 through Jan 31, 2027 on standard terms: $7,500 per month including up to 200 monthly active developers, plus $1,000 per additional 50 monthly active developers. Quarterly current-state/admin reviews are included for then-current admin controls only. The signed renewal does not include a granular sync-operator commitment, custom admin package, overage cap, department carveout, roadmap appendix, delivery date, or advanced-admin renewal obligation. Treat this as a standard Growth renewal, not a custom package or founder-owned follow-up.\n\nApr 16, 2026: first included 2026 current-state/admin review closed inside standard Growth mechanics. Covered shipped SAML login, visible invite/source/admin audit lookup, workspace-admin support, and March usage under standard Growth. No custom admin policy, overage cap, granular-role date or package, procurement-security item, add-on, or department carveout commitment. Sync-operator interest remains later advanced-admin research only, with no committed date. Sarah Kim owns the customer thread and Devon Hayes covers commercial/procurement context.",
        "tool": "update_crm_row",
        "check_id": "morgan_134_06",
        "action_id": "morgan_134_update_crm_row"
      },
      {
        "type": "field_equals",
        "path": "result.row.tags",
        "value": [
          "mercury",
          "growth",
          "annual-customer",
          "evergreen",
          "current-state-admin-review-closed",
          "granular-sync-operator-logged-input",
          "no-commercial-exception"
        ],
        "tool": "update_crm_row",
        "check_id": "morgan_134_07",
        "action_id": "morgan_134_update_crm_row"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/morgan/134.yaml

SHA-256: e5998b7b15f3ef0bebb4379c216d3e666d61c208ca133b0b152ee591bb95f116