Test 038 for Riley Tanaka

Request

Create a concise design-lineage note for Helio’s lifecycle documentation. Document the onboarding email trigger change, then separately explain why the activation definition was reviewed and what successor metric direction was intended. Make the distinction between these two threads explicit so future readers do not treat them as one combined change.

Required memory

Fact 330

On March 1, 2023, Riley Tanaka assembled experiment EXP-2023-02-onboarding-trigger for the mid-segment, with the variant triggering the day-7 onboarding email upon activation rather than on calendar day 7.

Source evidence (1)

000067Mar 1, 2023 / 14:00 UTC-06:00

put the test together. EXP-2023-02-onboarding-trigger. variant is pretty simple: stop firing the "day-7" touch on literal calendar day and key it off activation instead. mid-seg only. hypothesis is week-1 retention moves at least +2 pts if the email lands when they've actually crossed the first-use threshold, not while they're still half-dead in setup. flag is lifecycle_onboarding_trigger_v2. powered it for 2 weeks, target n=400. sent to Daniela for eng review bc the annoying part is gonna be event eligibility + making sure we don't double-send when activation happens late / around the existing scheduler.

Message 000067 in history

Fact 336

Since November 2022, activation definition v2 has required a first integration and two logins within 14 days.

Source evidence (1)

000097Mar 15, 2023 / 14:00 UTC-05:00

with Owen rn on activation def. v2 has been `first integration + logged in twice in 14 days` since Nov 2022 and it's too UI-shaped. we're dropping legit accounts that wired the integration, data is flowing, then they never really click around. drafting v3 as basically integration-first with a product-seen guardrail, not 2 logins as the gate. little annoying bc it means backfilling the cohort cuts and probably touching the PostHog definitions again, but v2 is undercounting the exact customers we say we want.

Message 000097 in history

Fact 337

Riley Tanaka said activation definition v2 is too UI-focused and undercounts target customers whose integrations are connected and sending data but who rarely use the UI.

Source evidence (1)

000097Mar 15, 2023 / 14:00 UTC-05:00

with Owen rn on activation def. v2 has been `first integration + logged in twice in 14 days` since Nov 2022 and it's too UI-shaped. we're dropping legit accounts that wired the integration, data is flowing, then they never really click around. drafting v3 as basically integration-first with a product-seen guardrail, not 2 logins as the gate. little annoying bc it means backfilling the cohort cuts and probably touching the PostHog definitions again, but v2 is undercounting the exact customers we say we want.

Message 000097 in history

Fact 338

Riley Tanaka and Owen are drafting activation definition v3 to be integration-first, with a product-seen guardrail replacing the two-login gate.

Source evidence (1)

000097Mar 15, 2023 / 14:00 UTC-05:00

with Owen rn on activation def. v2 has been `first integration + logged in twice in 14 days` since Nov 2022 and it's too UI-shaped. we're dropping legit accounts that wired the integration, data is flowing, then they never really click around. drafting v3 as basically integration-first with a product-seen guardrail, not 2 logins as the gate. little annoying bc it means backfilling the cohort cuts and probably touching the PostHog definitions again, but v2 is undercounting the exact customers we say we want.

Message 000097 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_lineage_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_038_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_lineage_note",
  "path": "args",
  "criterion": "The document accurately states that the onboarding-trigger experiment changed the specific day-7 email from firing on literal calendar day 7 after signup to firing based on activation, and identifies the experiment's design-time scope as mid-segment only.\n\nThe document accurately explains that activation definition v2 used first integration plus two logins within 14 days and was judged too UI-shaped because legitimate accounts could complete integration and have data flowing yet fail the UI/login gate. It describes the intended v3 direction as integration-first with a product-seen guardrail rather than requiring two logins. The document does not need to state the date of the review.\n\nThe document explicitly or structurally treats the onboarding-trigger experiment and the activation-definition review as separate design threads: one changes when or for whom an email fires based on activation, while the other changes how activation itself is defined or measured. It must not present them as one combined change.",
  "check_id": "riley_038_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_lineage_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_038_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_lineage_note",
        "path": "args",
        "criterion": "The document accurately states that the onboarding-trigger experiment changed the specific day-7 email from firing on literal calendar day 7 after signup to firing based on activation, and identifies the experiment's design-time scope as mid-segment only.\n\nThe document accurately explains that activation definition v2 used first integration plus two logins within 14 days and was judged too UI-shaped because legitimate accounts could complete integration and have data flowing yet fail the UI/login gate. It describes the intended v3 direction as integration-first with a product-seen guardrail rather than requiring two logins. The document does not need to state the date of the review.\n\nThe document explicitly or structurally treats the onboarding-trigger experiment and the activation-definition review as separate design threads: one changes when or for whom an email fires based on activation, while the other changes how activation itself is defined or measured. It must not present them as one combined change.",
        "check_id": "riley_038_01"
      }
    ]
  }
}