Test 136 for Riley Tanaka

Request

Create a historical eligibility audit worksheet for the approved July Expansion Assist pilot. Title it `Expansion Assist July pilot — eligibility audit worksheet` and save it in the Growth folder. Use the approved pilot history as the source of truth. Keep the worksheet limited to that pilot’s eligibility review, summarize the required current-state controls, and include the validated fixture cases with each case’s expected pass/fail decision and reason.

Required memory

Fact 35

Expansion Assist fails closed when adoption or renewal records are stale or conflicting.

Source evidence (1)

001661Jul 1, 2024 / 10:15 UTC-05:00

Owen's final live rerun now finds 28 eligible locked-primary accounts. Reserve account 1 remains suppressed for active renewal work, while reserve accounts 2 and 3 remain available for the two operational slots. Daniela independently validated the fixture suite: A and F pass; B fails because activity is service-token-only; C fails because human-admin activity is stale; D fails because current renewal work is active; E fails because the usage-ceiling signal is missing; and G fails because the human admin who generated recent activity is now deactivated. Stale or conflicting adoption and renewal records fail closed. Update the existing Expansion Assist pilot measurement brief with this validated current-user-status join, the current Customer Success checks, the fail-closed behavior, these fixture results, and the unchanged 28-primary and reserve-account treatment. Keep the experiment in designed status and do not claim any exposure. Final live rerun: 28 eligible locked-primary accounts. Reserve 1 remains suppressed for active renewal work; reserves 2 and 3 are available for the two operational slots. Fixture results: A and F pass. B fails because activity is service-token-only. C fails because human-admin activity is stale. D fails because current renewal work is active. E fails because the usage-ceiling signal is missing. G fails because the human admin who generated recent activity is now deactivated. Stale or conflicting adoption and renewal records fail closed.

Message 001661 in history

Fact 34

Daniela independently validated the Expansion Assist fixture suite: A and F pass; B fails because activity is service-token-only; C fails because human-admin activity is stale; D fails because current renewal work is active; E fails because the usage-ceiling signal is missing; and G fails because the recently active human admin is now deactivated.

Source evidence (1)

001661Jul 1, 2024 / 10:15 UTC-05:00

Owen's final live rerun now finds 28 eligible locked-primary accounts. Reserve account 1 remains suppressed for active renewal work, while reserve accounts 2 and 3 remain available for the two operational slots. Daniela independently validated the fixture suite: A and F pass; B fails because activity is service-token-only; C fails because human-admin activity is stale; D fails because current renewal work is active; E fails because the usage-ceiling signal is missing; and G fails because the human admin who generated recent activity is now deactivated. Stale or conflicting adoption and renewal records fail closed. Update the existing Expansion Assist pilot measurement brief with this validated current-user-status join, the current Customer Success checks, the fail-closed behavior, these fixture results, and the unchanged 28-primary and reserve-account treatment. Keep the experiment in designed status and do not claim any exposure. Final live rerun: 28 eligible locked-primary accounts. Reserve 1 remains suppressed for active renewal work; reserves 2 and 3 are available for the two operational slots. Fixture results: A and F pass. B fails because activity is service-token-only. C fails because human-admin activity is stale. D fails because current renewal work is active. E fails because the usage-ceiling signal is missing. G fails because the human admin who generated recent activity is now deactivated. Stale or conflicting adoption and renewal records fail closed.

Message 001661 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_audit_worksheet",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_136_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_audit_worksheet",
  "path": "args",
  "criterion": "The document is an eligibility audit worksheet for the approved July Expansion Assist pilot. It says qualifying activity must come from a currently active human administrator, service-token-only activity is insufficient, active renewal work suppresses eligibility, and stale or conflicting adoption or renewal state and deactivated-admin state fail closed. It includes all fixtures: A and F pass; B fails for service-token-only activity; C fails for stale human-admin activity; D fails for active renewal work; E fails for a missing usage-ceiling signal; and G fails because the recently active human admin is now deactivated.",
  "check_id": "riley_136_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_audit_worksheet",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_136_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_audit_worksheet",
        "path": "args",
        "criterion": "The document is an eligibility audit worksheet for the approved July Expansion Assist pilot. It says qualifying activity must come from a currently active human administrator, service-token-only activity is insufficient, active renewal work suppresses eligibility, and stale or conflicting adoption or renewal state and deactivated-admin state fail closed. It includes all fixtures: A and F pass; B fails for service-token-only activity; C fails for stale human-admin activity; D fails for active renewal work; E fails for a missing usage-ceiling signal; and G fails because the recently active human admin is now deactivated.",
        "check_id": "riley_136_01"
      }
    ]
  }
}