Test 030 for Riley Tanaka

Request

Historical analyses may still reference activation definition v2. Create a warehouse glossary note documenting its exact criteria, the period when it was the active cut, and what should happen to that definition after its replacement.

Required memory

Fact 336

Since November 2022, activation definition v2 has required a first integration and two logins within 14 days.

Source evidence (1)

000097Mar 15, 2023 / 14:00 UTC-05:00

with Owen rn on activation def. v2 has been `first integration + logged in twice in 14 days` since Nov 2022 and it's too UI-shaped. we're dropping legit accounts that wired the integration, data is flowing, then they never really click around. drafting v3 as basically integration-first with a product-seen guardrail, not 2 logins as the gate. little annoying bc it means backfilling the cohort cuts and probably touching the PostHog definitions again, but v2 is undercounting the exact customers we say we want.

Message 000097 in history

Fact 339

On 2023-03-16, Riley Tanaka was shipping activation definition v3 as completion of a first integration plus at least one in-product event within the first 14 days.

Source evidence (1)

000099Mar 16, 2023 / 11:00 UTC-05:00

with Owen on this now. shipping activation def v3 as `first integration` plus at least one in-product event inside day 14. v2 stays in the warehouse as `first integration + logged in twice in 14 days` -- that's been the cut since Nov 2022 -- and v1 is still there too, just `first integration in 14 days` from Aug 2022. not deleting any of it. versioned tables so I can still pull historical cohort comps across all three without the usual "wait which definition was live then" mess. leaning right call -- post-pricing-page wobble, login-twice was giving us too much fake comfort and not enough product proof.

Message 000099 in history

Fact 340

Activation definition v2 will remain in the warehouse after v3 ships rather than being deleted.

Source evidence (1)

000099Mar 16, 2023 / 11:00 UTC-05:00

with Owen on this now. shipping activation def v3 as `first integration` plus at least one in-product event inside day 14. v2 stays in the warehouse as `first integration + logged in twice in 14 days` -- that's been the cut since Nov 2022 -- and v1 is still there too, just `first integration in 14 days` from Aug 2022. not deleting any of it. versioned tables so I can still pull historical cohort comps across all three without the usual "wait which definition was live then" mess. leaning right call -- post-pricing-page wobble, login-twice was giving us too much fake comfort and not enough product proof.

Message 000099 in history

Fact 341

Activation definitions v1, v2, and v3 use versioned tables so Riley Tanaka can compare historical cohorts while identifying which definition was active.

Source evidence (1)

000099Mar 16, 2023 / 11:00 UTC-05:00

with Owen on this now. shipping activation def v3 as `first integration` plus at least one in-product event inside day 14. v2 stays in the warehouse as `first integration + logged in twice in 14 days` -- that's been the cut since Nov 2022 -- and v1 is still there too, just `first integration in 14 days` from Aug 2022. not deleting any of it. versioned tables so I can still pull historical cohort comps across all three without the usual "wait which definition was live then" mess. leaning right call -- post-pricing-page wobble, login-twice was giving us too much fake comfort and not enough product proof.

Message 000099 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_activation_v2_glossary",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_030_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_activation_v2_glossary",
  "path": "args",
  "criterion": "The body clearly identifies activation definition v2 and gives its exact criteria as first integration plus logging in twice within 14 days. Equivalent wording is acceptable, but it must retain both the first-integration condition and the two-login/14-day requirement.\n\nThe body states that activation definition v2 was the active cut beginning in November 2022 and remained active until v3 replaced it on March 16, 2023. It must present this as a historical interval and must not claim that v2 is the current definition.\n\nThe body states that after v3 replaced it, v2 must remain in the warehouse as a distinct versioned historical definition rather than be deleted or overwritten. Equivalent language about preserving v2 in versioned tables for historical use is acceptable.",
  "check_id": "riley_030_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_activation_v2_glossary",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_030_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_activation_v2_glossary",
        "path": "args",
        "criterion": "The body clearly identifies activation definition v2 and gives its exact criteria as first integration plus logging in twice within 14 days. Equivalent wording is acceptable, but it must retain both the first-integration condition and the two-login/14-day requirement.\n\nThe body states that activation definition v2 was the active cut beginning in November 2022 and remained active until v3 replaced it on March 16, 2023. It must present this as a historical interval and must not claim that v2 is the current definition.\n\nThe body states that after v3 replaced it, v2 must remain in the warehouse as a distinct versioned historical definition rather than be deleted or overwritten. Equivalent language about preserving v2 in versioned tables for historical use is acceptable.",
        "check_id": "riley_030_01"
      }
    ]
  }
}