Test 033 for Riley Tanaka

Request

Create a historical design record for the original expansion-nudge experiment that includes its experiment identifier, initial rollout, cohort scope, feature flag, and intended powered window. Clearly label it as the original pre-launch plan so it isn't confused with later operation.

Required memory

Fact 384

On May 21, 2023, Riley Tanaka defined experiment EXP-2023-05-expansion-nudge to show an in-product prompt on day 60 to retained mid-segment v2 accounts with 75–200 seats, targeting at least $8 in additional expansion MRR per account.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Fact 385

EXP-2023-05-expansion-nudge will use feature flag `expansion_nudge_60d_v1` and begin at 25% rollout to avoid broad exposure before validating the prompt's copy and placement.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Fact 386

EXP-2023-05-expansion-nudge has a planned two-week window, conditional on the power analysis supporting it.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_historical_design_record",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_033_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_historical_design_record",
  "path": "args",
  "criterion": "Considering the document title and body together, the record must identify the original expansion-nudge design as EXP-2023-05-expansion-nudge and include all of these plan parameters: a 25% starting rollout; scope limited to retained mid-seg v2 accounts in the 75–200-seat band; feature flag `expansion_nudge_60d_v1`; and an intended two-week powered window. Equivalent wording is allowed, but the experiment identifier and feature flag must be exact.\n\nConsidering the title and body together, the record must clearly and explicitly label these values as the original pre-launch plan or pre-launch baseline rather than current operating state, and it must not state or imply that they are current.",
  "check_id": "riley_033_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_historical_design_record",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_033_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_historical_design_record",
        "path": "args",
        "criterion": "Considering the document title and body together, the record must identify the original expansion-nudge design as EXP-2023-05-expansion-nudge and include all of these plan parameters: a 25% starting rollout; scope limited to retained mid-seg v2 accounts in the 75–200-seat band; feature flag `expansion_nudge_60d_v1`; and an intended two-week powered window. Equivalent wording is allowed, but the experiment identifier and feature flag must be exact.\n\nConsidering the title and body together, the record must clearly and explicitly label these values as the original pre-launch plan or pre-launch baseline rather than current operating state, and it must not state or imply that they are current.",
        "check_id": "riley_033_01"
      }
    ]
  }
}