Test 191

Jan 1, 2028 / 2 facts

Request

Please create a concise historical launch-scope card for EXP-2023-02-onboarding-trigger. Include the launch date, exact feature-flag key, initial allocation, and cohort definition used at launch. Make it unmistakable that this card documents only the initial experiment configuration, rather than the later rollout or the newer default cohort definition.

Required memory

Fact 334

Riley Tanaka launched EXP-2023-02-onboarding-trigger on March 13, 2023, one day later than planned because of engineering review, with `lifecycle_onboarding_trigger_v2` at 50% for the mid-segment 50–200-seat v1 bucket.

Source evidence (1)

000092Mar 13, 2023 / 10:00 UTC-05:00

rolled EXP-2023-02-onboarding-trigger this morning. `lifecycle_onboarding_trigger_v2` at 50% on mid-seg only -- the 50-200 seat v1 bucket. eng review pushed it a day so whatever, Mar 13 not Mar 12. target n is still 400 and i want the full 2 weeks before anybody starts storytelling off noise. dropped the launch note and tagged Priya, Daniela, Owen, Ines. also called out in the doc that this one is scope-disjoint from `pricing_page_2023q2_rev` -- different cohort, different metric-of-record, so if somebody tries to mash attribution together later i can just point at the note.

Message 000092 in history

Fact 359

Riley Tanaka published the launch note for the `lifecycle_onboarding_trigger_v2` rollout.

Source evidence (1)

000165Apr 10, 2023 / 10:00 UTC-05:00

ok shipped. `lifecycle_onboarding_trigger_v2` is at 100% in PostHog now, confirmed, and I put the launch note up. scope is all mid-segment + growth tier, not starter. first post-ship cohort started this morning so now it's basically boring watch-the-curve work for a few weeks. Ines did one last pass on the email/template stuff before flip -- token fallbacks, locale branches, footer/unsub on the old Customer.io paths -- nothing weird. leaning not to touch anything unless Owen's pull shows some dumb mix shift by seat band or source.

Message 000165 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_launch_scope_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_191_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_launch_scope_card",
  "path": "args",
  "criterion": "The document accurately identifies EXP-2023-02-onboarding-trigger and states all four initial launch details: launch date March 13, 2023; exact flag key `lifecycle_onboarding_trigger_v2`; initial allocation 50%; and the mid-segment v1 cohort of 50–200 seats. Missing, altered, or contradicted details must fail.\n\nThe document clearly labels the stated scope as the initial experiment or launch configuration only and prevents readers from treating it as the later rollout scope or the newer/default cohort definition. It need not state the later rollout values.",
  "check_id": "riley_191_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_launch_scope_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_191_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_launch_scope_card",
        "path": "args",
        "criterion": "The document accurately identifies EXP-2023-02-onboarding-trigger and states all four initial launch details: launch date March 13, 2023; exact flag key `lifecycle_onboarding_trigger_v2`; initial allocation 50%; and the mid-segment v1 cohort of 50–200 seats. Missing, altered, or contradicted details must fail.\n\nThe document clearly labels the stated scope as the initial experiment or launch configuration only and prevents readers from treating it as the later rollout scope or the newer/default cohort definition. It need not state the later rollout values.",
        "check_id": "riley_191_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/191.yaml

SHA-256: eb38e1ee5a3f0447625ff091f4100b1a0ca0d75b500e0cbe4ae75fc810b8687d