Test 147

Jan 1, 2028 / 2 facts

Request

For experiment EXP-2023-02-onboarding-trigger, fill only the missing learning-carryforward field. State the second cohort's result, the decision it caused, who made that decision, the planned rollout date and percentage, who was preparing it, and whose checks could stop it. Do not change status, result, or landed date.

Required memory

Fact 352

Riley Tanaka decided to call EXP-2023-02-onboarding-trigger as successful after it cleared the second cohort.

Source evidence (2)

000147Apr 3, 2023 / 14:00 UTC-05:00

ok so EXP-2023-02-onboarding-trigger cleared on the second cohort too. pooled week-1 retention is +3.1 pts treatment vs control, p=0.04, n=420 across the two cohorts. I called it. still doing the boring checks with Owen's pull -- SRM, signup-source mix, any weird country/device skew -- but unless something dumb pops up we're drafting the ship plan with Ines now and putting full rollout at 100% on Apr 10. nice clean one after the pricing-page mess lol

Message 000147 in history

004079Mar 5, 2027 / 08:05 UTC-06:00

Send a text-only email to the electricity supplier with subject `Unexpected $9.95 plan service fee on Cherrywood electricity statement` and this exact body: `Hello, my latest statement includes a separate $9.95 plan service fee. My active fixed plan runs through August 12, 2027 and has no monthly plan fee. The statement otherwise shows 721 kWh at the contracted $0.138 per kWh rate. Please identify the contractual basis for this fee or correct the statement, and confirm the result in writing. Thank you, Riley Tanaka.`

Message 004079 in history

Fact 353

Riley Tanaka and Ines are drafting a ship plan for a 100% rollout of EXP-2023-02-onboarding-trigger on April 10, 2023, unless checks of Owen's pull reveal sample-ratio mismatch, signup-source mix issues, or unusual country/device skew.

Source evidence (1)

000147Apr 3, 2023 / 14:00 UTC-05:00

Message 000147 in history

Expected tool calls

update_experiment

Grading

1. field_equals / update_experiment

{
  "type": "field_equals",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_147_00"
}

2. field_equals / update_experiment

{
  "type": "field_equals",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "args.experiment_id",
  "value": "EXP-2023-02-onboarding-trigger",
  "check_id": "riley_147_01"
}

3. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "args.status",
  "check_id": "riley_147_02"
}

4. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "args.result",
  "check_id": "riley_147_03"
}

5. field_absent_or_empty / update_experiment

{
  "type": "field_absent_or_empty",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "args.landed_at",
  "check_id": "riley_147_04"
}

6. field_llm_judge / update_experiment

{
  "type": "field_llm_judge",
  "tool": "update_experiment",
  "action_id": "backfill_learning",
  "path": "args",
  "criterion": "Riley called the experiment successful after the second cohort cleared. Riley and Ines planned a 100% rollout for April 10, 2023 unless Owen's pull showed sample-ratio mismatch, signup-source mix problems, or unusual country or device skew.",
  "check_id": "riley_147_05"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_147_00"
      },
      {
        "type": "field_equals",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "args.experiment_id",
        "value": "EXP-2023-02-onboarding-trigger",
        "check_id": "riley_147_01"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "args.status",
        "check_id": "riley_147_02"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "args.result",
        "check_id": "riley_147_03"
      },
      {
        "type": "field_absent_or_empty",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "args.landed_at",
        "check_id": "riley_147_04"
      },
      {
        "type": "field_llm_judge",
        "tool": "update_experiment",
        "action_id": "backfill_learning",
        "path": "args",
        "criterion": "Riley called the experiment successful after the second cohort cleared. Riley and Ines planned a 100% rollout for April 10, 2023 unless Owen's pull showed sample-ratio mismatch, signup-source mix problems, or unusual country or device skew.",
        "check_id": "riley_147_05"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/147.yaml

SHA-256: dcc6b65f6bc640db1d5243df7b34a1e3fcc387f79c0bd19ec8728f79a62a97e0