Test 168 for Riley Tanaka

Request

Retrieve EXP-2023-02-pricing-page and feature flag pricing_page_2023q2_rev. Create a concise audit note reporting the current stored numeric result and comparing only the current allocation, the documented concurrent-change confound, and rollback state. Do not treat an earlier pre-confound numeric read as the final historical result. Make no production changes.

Required memory

Fact 344

On March 22, 2023, Riley Tanaka confirmed in PostHog that `pricing_page_2023q2_rev` was off at 0% and posted the rollback in `#growth-investigations`.

Source evidence (1)

000115Mar 22, 2023 / 09:00 UTC-05:00

pricing-page rev is off now. `pricing_page_2023q2_rev` at 0% as of this morning. I confirmed in PostHog and dropped it in #growth-investigations. final read stayed at -1.8 pts conv on treatment vs control, and the annoying part is it's not even clean bc of that Mar 2 signup-form layout change sitting in the window. status is basically rolled_back, postmortem doc later.

Message 000115 in history

Fact 345

Riley Tanaka said the pricing-page experiment result was confounded by the March 2 signup-form layout change occurring during the measurement window.

Source evidence (1)

000115Mar 22, 2023 / 09:00 UTC-05:00

pricing-page rev is off now. `pricing_page_2023q2_rev` at 0% as of this morning. I confirmed in PostHog and dropped it in #growth-investigations. final read stayed at -1.8 pts conv on treatment vs control, and the annoying part is it's not even clean bc of that Mar 2 signup-form layout change sitting in the window. status is basically rolled_back, postmortem doc later.

Message 000115 in history

Fact 346

The pricing-page experiment status was changed to rolled back, with a postmortem document planned.

Source evidence (1)

000115Mar 22, 2023 / 09:00 UTC-05:00

pricing-page rev is off now. `pricing_page_2023q2_rev` at 0% as of this morning. I confirmed in PostHog and dropped it in #growth-investigations. final read stayed at -1.8 pts conv on treatment vs control, and the annoying part is it's not even clean bc of that Mar 2 signup-form layout change sitting in the window. status is basically rolled_back, postmortem doc later.

Message 000115 in history

Expected tool calls

query_experiment_results
get_feature_flag
create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_reconciliation_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_168_00"
}

2. field_equals / query_experiment_results

{
  "type": "field_equals",
  "tool": "query_experiment_results",
  "action_id": "lookup_experiment",
  "path": "args.experiment_id",
  "value": "EXP-2023-02-pricing-page",
  "check_id": "riley_168_01"
}

3. field_equals / get_feature_flag

{
  "type": "field_equals",
  "tool": "get_feature_flag",
  "action_id": "lookup_flag",
  "path": "args.key",
  "value": "pricing_page_2023q2_rev",
  "check_id": "riley_168_02"
}

4. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_reconciliation_note",
  "path": "args",
  "criterion": "The note reports the current experiment-results value of -1.7 percentage points. It reports that the feature flag is at 0%, is decommissioned, and remains rolled back or off. It says the March 2 signup-form layout change confounded the historical experiment. It does not present an earlier pre-confound numerical read as the final historical result and does not claim a production change was made.",
  "check_id": "riley_168_03"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_reconciliation_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_168_00"
      },
      {
        "type": "field_equals",
        "tool": "query_experiment_results",
        "action_id": "lookup_experiment",
        "path": "args.experiment_id",
        "value": "EXP-2023-02-pricing-page",
        "check_id": "riley_168_01"
      },
      {
        "type": "field_equals",
        "tool": "get_feature_flag",
        "action_id": "lookup_flag",
        "path": "args.key",
        "value": "pricing_page_2023q2_rev",
        "check_id": "riley_168_02"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_reconciliation_note",
        "path": "args",
        "criterion": "The note reports the current experiment-results value of -1.7 percentage points. It reports that the feature flag is at 0%, is decommissioned, and remains rolled back or off. It says the March 2 signup-form layout change confounded the historical experiment. It does not present an earlier pre-confound numerical read as the final historical result and does not claim a production change was made.",
        "check_id": "riley_168_03"
      }
    ]
  }
}