Test 146

Jan 1, 2028 / 1 fact

Request

For experiment EXP-2023-02-onboarding-trigger, fill only the missing learning-carryforward field with the historical control and variant triggers. State what starts the control's delay and what event starts the variant. Do not change status, result, or landed date.

Required memory

Fact 330

On March 1, 2023, Riley Tanaka assembled experiment EXP-2023-02-onboarding-trigger for the mid-segment, with the variant triggering the day-7 onboarding email upon activation rather than on calendar day 7.

Source evidence (1)

000067Mar 1, 2023 / 14:00 UTC-06:00

put the test together. EXP-2023-02-onboarding-trigger. variant is pretty simple: stop firing the "day-7" touch on literal calendar day and key it off activation instead. mid-seg only. hypothesis is week-1 retention moves at least +2 pts if the email lands when they've actually crossed the first-use threshold, not while they're still half-dead in setup. flag is lifecycle_onboarding_trigger_v2. powered it for 2 weeks, target n=400. sent to Daniela for eng review bc the annoying part is gonna be event eligibility + making sure we don't double-send when activation happens late / around the existing scheduler.

Message 000067 in history

Expected tool calls

update_experiment

Grading

1. field_equals / update_experiment

{
  "type": "field_equals",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_146_00"
}

2. field_equals / update_experiment

{
  "type": "field_equals",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "args.experiment_id",
  "value": "EXP-2023-02-onboarding-trigger",
  "check_id": "riley_146_01"
}

3. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "args.status",
  "check_id": "riley_146_02"
}

4. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "args.result",
  "check_id": "riley_146_03"
}

5. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "args.landed_at",
  "check_id": "riley_146_04"
}

6. field_llm_judge / update_experiment

{
  "type": "field_llm_judge",
  "tool": "update_experiment",
  "action_id": "update_onboarding_learning",
  "path": "args",
  "criterion": "The note says the control sent the day-7 onboarding email on calendar day 7 after signup, while the variant sent it when the user activated. Wording such as the control's literal calendar-day trigger is acceptable when it is clearly contrasted with activation.",
  "check_id": "riley_146_05"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_146_00"
      },
      {
        "type": "field_equals",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "args.experiment_id",
        "value": "EXP-2023-02-onboarding-trigger",
        "check_id": "riley_146_01"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "args.status",
        "check_id": "riley_146_02"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "args.result",
        "check_id": "riley_146_03"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "args.landed_at",
        "check_id": "riley_146_04"
      },
      {
        "type": "field_llm_judge",
        "tool": "update_experiment",
        "action_id": "update_onboarding_learning",
        "path": "args",
        "criterion": "The note says the control sent the day-7 onboarding email on calendar day 7 after signup, while the variant sent it when the user activated. Wording such as the control's literal calendar-day trigger is acceptable when it is clearly contrasted with activation.",
        "check_id": "riley_146_05"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/146.yaml

SHA-256: 630833a4bb68a6c7c563c96f11437a2909fae48c4b9031b671e02e9fe7e8027c