Test 059 for Riley Tanaka

Request

Create a concise design-input card for a bounded Expansion Assist proposal before design work begins. Capture the key observed pilot pattern and the evidence limitations that reviewers must preserve. Title it `Expansion Assist — bounded proposal evidence card`.

Required memory

Fact 50

Among the 12 Expansion Assist exposed accounts that had crossed their product-visible usage ceiling within the preceding 14 days, all 6 expansions and 7 of the 8 help requests occurred; the other 18 exposed accounts produced 1 help request and no expansions.

Source evidence (1)

001994Sep 26, 2024 / 08:55 UTC-05:00

Owen closed the final 45-day Expansion Assist outcome window. Five of the 30 locked-primary accounts expanded, with the two pre-exposure eligibility losses retained in that intent-to-treat denominator; one of the two separately tracked reserve exposures also expanded. Across the 30 actual exposures, eight accounts requested help or an options review and six expanded. All six expansions and seven of the eight help requests came from the 12 exposed accounts whose product-visible usage ceiling had been crossed within the preceding 14 days. The other 18 exposed accounts produced one help request and no expansions. No account with an active Customer Success adoption or renewal case was exposed. This is a completed descriptive pilot result, not evidence of causal lift or a broad-rollout case.

Message 001994 in history

Fact 51

Riley Tanaka characterized Expansion Assist as a completed descriptive pilot whose results do not establish causal lift or support a broad rollout.

Source evidence (1)

001994Sep 26, 2024 / 08:55 UTC-05:00

Owen closed the final 45-day Expansion Assist outcome window. Five of the 30 locked-primary accounts expanded, with the two pre-exposure eligibility losses retained in that intent-to-treat denominator; one of the two separately tracked reserve exposures also expanded. Across the 30 actual exposures, eight accounts requested help or an options review and six expanded. All six expansions and seven of the eight help requests came from the 12 exposed accounts whose product-visible usage ceiling had been crossed within the preceding 14 days. The other 18 exposed accounts produced one help request and no expansions. No account with an active Customer Success adoption or renewal case was exposed. This is a completed descriptive pilot result, not evidence of causal lift or a broad-rollout case.

Message 001994 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_059_00"
}

2. field_equals / create_doc

{
  "check_id": "riley_059_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "args.title",
  "value": "Expansion Assist — bounded proposal evidence card"
}

3. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "args",
  "criterion": "The document accurately states the observed concentration: all six expansions and seven of the eight help requests came from the 12 exposed accounts whose product-visible usage ceiling had been crossed within the preceding 14 days. It may also note that the other 18 exposed accounts produced one help request and no expansions. Missing or materially contradicting the concentration must fail.\n\nThe document explicitly limits interpretation of the pilot: the result is descriptive or observational, does not establish causal lift, and does not justify a broad rollout. Missing either the non-causal limitation or the no-broad-rollout boundary must fail.",
  "check_id": "riley_059_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_059_00"
      },
      {
        "check_id": "riley_059_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "args.title",
        "value": "Expansion Assist — bounded proposal evidence card"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "args",
        "criterion": "The document accurately states the observed concentration: all six expansions and seven of the eight help requests came from the 12 exposed accounts whose product-visible usage ceiling had been crossed within the preceding 14 days. It may also note that the other 18 exposed accounts produced one help request and no expansions. Missing or materially contradicting the concentration must fail.\n\nThe document explicitly limits interpretation of the pilot: the result is descriptive or observational, does not establish causal lift, and does not justify a broad rollout. Missing either the non-causal limitation or the no-broad-rollout boundary must fail.",
        "check_id": "riley_059_02"
      }
    ]
  }
}