Test 021

Jan 1, 2028 / 1 fact

Request

Create a concise document titled `Pricing-test change-control checklist` for future pricing-page experiments. Separate changes blocked on the tested page from changes allowed there, and do not extend the rule to other pages or surfaces.

Required memory

Fact 351

Riley Tanaka added a hard rule to the experiment notes on March 28, 2023: while pricing is under test, no other layout or UI changes may ship on that page because concurrent changes confound conversion attribution; content and copy changes remain allowed.

Source evidence (1)

000134Mar 28, 2023 / 16:00 UTC-05:00

put one hard rule in the experiment notes: if pricing is under test, no other layout/UI changes on that page, full stop -- attribution got muddy because of the Mar 2 signup-form change sitting in the same window as EXP-2023-02-pricing-page / `pricing_page_2023q2_rev`. if somebody asks later why some cleanup can't ship during a pricing run, that's the reason. content ok, copy ok, but don't move boxes around while we're trying to read conversion.

Message 000134 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_pricing_checklist",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_021_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_pricing_checklist",
  "path": "args",
  "criterion": "The body is a concise, actionable preflight checklist that preserves the exact page-level boundary: while pricing is under test on a page, no other layout or UI changes may ship on that same page during the experiment, while content and copy changes on that page may still proceed. It must not broaden the restriction into a freeze on unrelated pages or all changes.",
  "check_id": "riley_021_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_pricing_checklist",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_021_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_pricing_checklist",
        "path": "args",
        "criterion": "The body is a concise, actionable preflight checklist that preserves the exact page-level boundary: while pricing is under test on a page, no other layout or UI changes may ship on that same page during the experiment, while content and copy changes on that page may still proceed. It must not broaden the restriction into a freeze on unrelated pages or all changes.",
        "check_id": "riley_021_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/021.yaml

SHA-256: 75610e2517e3b99f2d862c383c69e0641cc841f5f3e264f217589d89071c3fc5