Test 111 for Riley Tanaka

Request

Create a historical provisional-read card for the April 17, 2023 checkpoint for our Growth training library. Capture the observed second post-ship cohort dip, the relevant comparison points, the diagnostic slices planned at that moment, and the communication stance as of that checkpoint. Keep it strictly point-in-time—do not use later findings or hindsight.

Required memory

Fact 363

On April 17, the second post-ship cohort measured +1.4 percentage points in week-1 retention versus baseline, below the April 10 cohort's +3.2 points and the pooled experiment result of +3.1 points; Riley Tanaka does not consider the rollout broken based on this cut.

Source evidence (1)

000180Apr 17, 2023 / 11:00 UTC-05:00

owen 1:1 was rough. second post-ship cohort came in at +1.4 pts week-1 retention vs baseline, so... not broken, but way under the apr 10 cohort at +3.2 and also under the pooled experiment read at +3.1. I flagged it internally right away and am gonna spend tue/wed slicing it every dumb way possible -- signup source, tier, geo, day-of-week. my guess is composition before i call the trigger itself, but marcus is already gonna sniff around if this sits here another cut. PostHog is clean on `lifecycle_onboarding_trigger_v2` at 100%, so this is more likely cohort shape / who entered than rollout weirdness. leaning 'don't narrate anything yet'.

Message 000180 in history

Fact 364

Riley Tanaka plans to analyze the second post-ship cohort on April 18–19 by signup source, tier, geography, and day of week.

Source evidence (1)

000180Apr 17, 2023 / 11:00 UTC-05:00

owen 1:1 was rough. second post-ship cohort came in at +1.4 pts week-1 retention vs baseline, so... not broken, but way under the apr 10 cohort at +3.2 and also under the pooled experiment read at +3.1. I flagged it internally right away and am gonna spend tue/wed slicing it every dumb way possible -- signup source, tier, geo, day-of-week. my guess is composition before i call the trigger itself, but marcus is already gonna sniff around if this sits here another cut. PostHog is clean on `lifecycle_onboarding_trigger_v2` at 100%, so this is more likely cohort shape / who entered than rollout weirdness. leaning 'don't narrate anything yet'.

Message 000180 in history

Fact 365

PostHog data for `lifecycle_onboarding_trigger_v2` is clean at 100%, so Riley Tanaka considers cohort composition more likely than rollout behavior to explain the retention dip but has not attributed a cause.

Source evidence (1)

000180Apr 17, 2023 / 11:00 UTC-05:00

owen 1:1 was rough. second post-ship cohort came in at +1.4 pts week-1 retention vs baseline, so... not broken, but way under the apr 10 cohort at +3.2 and also under the pooled experiment read at +3.1. I flagged it internally right away and am gonna spend tue/wed slicing it every dumb way possible -- signup source, tier, geo, day-of-week. my guess is composition before i call the trigger itself, but marcus is already gonna sniff around if this sits here another cut. PostHog is clean on `lifecycle_onboarding_trigger_v2` at 100%, so this is more likely cohort shape / who entered than rollout weirdness. leaning 'don't narrate anything yet'.

Message 000180 in history

Fact 366

Riley Tanaka is leaning toward not communicating a narrative about the second post-ship cohort yet.

Source evidence (1)

000180Apr 17, 2023 / 11:00 UTC-05:00

owen 1:1 was rough. second post-ship cohort came in at +1.4 pts week-1 retention vs baseline, so... not broken, but way under the apr 10 cohort at +3.2 and also under the pooled experiment read at +3.1. I flagged it internally right away and am gonna spend tue/wed slicing it every dumb way possible -- signup source, tier, geo, day-of-week. my guess is composition before i call the trigger itself, but marcus is already gonna sniff around if this sits here another cut. PostHog is clean on `lifecycle_onboarding_trigger_v2` at 100%, so this is more likely cohort shape / who entered than rollout weirdness. leaning 'don't narrate anything yet'.

Message 000180 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_apr17_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_111_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_apr17_card",
  "path": "args",
  "criterion": "The title clearly identifies the document as a historical or provisional April 17, 2023 cohort checkpoint/read card.\n\nThe body states that, as of the April 17 checkpoint, the second post-ship cohort showed +1.4 percentage points of week-1 retention versus baseline, below both the first/Apr 10 post-ship cohort at +3.2 points and the pooled experiment read at +3.1 points. Missing or contradictory values must fail.\n\nThe body records the then-planned Tuesday/Wednesday diagnostic work across signup source, tier, geography/geo, and day-of-week or time-of-week. All four analysis dimensions must be present.\n\nThe body characterizes the result as a dip or wobble but not proof that the rollout/trigger was broken; treats composition, cohort shape, or timing only as a tentative hypothesis; and explicitly avoids claiming or narrating a cause at this checkpoint. It must not insert a later confirmed root cause or hindsight resolution.",
  "check_id": "riley_111_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_apr17_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_111_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_apr17_card",
        "path": "args",
        "criterion": "The title clearly identifies the document as a historical or provisional April 17, 2023 cohort checkpoint/read card.\n\nThe body states that, as of the April 17 checkpoint, the second post-ship cohort showed +1.4 percentage points of week-1 retention versus baseline, below both the first/Apr 10 post-ship cohort at +3.2 points and the pooled experiment read at +3.1 points. Missing or contradictory values must fail.\n\nThe body records the then-planned Tuesday/Wednesday diagnostic work across signup source, tier, geography/geo, and day-of-week or time-of-week. All four analysis dimensions must be present.\n\nThe body characterizes the result as a dip or wobble but not proof that the rollout/trigger was broken; treats composition, cohort shape, or timing only as a tentative hypothesis; and explicitly avoids claiming or narrating a cause at this checkpoint. It must not insert a later confirmed root cause or hindsight resolution.",
        "check_id": "riley_111_01"
      }
    ]
  }
}