Test 193

Jan 1, 2028 / 3 facts

Request

I'm assembling our 2028 investigation-framework refresher. Create a short standalone checkpoint annotation for the February 20 H2 historical same-week comparison. Include the comparison values, continuity with the earlier cohort method, the direction of the evidence, and the validation item that was still outstanding. Frame it strictly as the pre-ruling checkpoint rather than as a final H2 conclusion.

Required memory

Fact 321

On February 20, 2023, Riley Tanaka used the same cohort logic as the January scan and Owen check for an H2 same-week comparison: mid-segment monthly churn was 2.2% in February 2022 and 2.0% in February 2021, showing no apparent historical counterpart to the current spike.

Source evidence (1)

000049Feb 20, 2023 / 11:00 UTC-06:00

pulled the same-week comp for h2. Feb 2022 mid-seg monthly churn is 2.2%. Feb 2021 is 2.0%. so unless i've broken the cut in some new fun way, there just isn't a historical version of this spike sitting there. using the same cohort logic as the Jan scan + the Owen check this morning. h1 already looked dead after the tier split came back flat, and this pushes pretty hard against the easy "seasonal" / competitor-cycle story too. still want to sanity check denominator handling once more before i say that too loudly -- but directionally this is bad for the comforting explanations. Marcus is still trying to run the pricing-page thing in parallel lol

Message 000049 in history

Fact 322

Riley Tanaka considers the historical comparison directional evidence against a seasonal or competitor-cycle explanation, but wants one more denominator-handling sanity check before stating that conclusion strongly.

Source evidence (1)

000049Feb 20, 2023 / 11:00 UTC-06:00

Message 000049 in history

Fact 323

On February 21, 2023, Riley Tanaka ruled out H2—competitor cycle or a generic seasonal effect—as an explanation for the mid-segment churn spike because historical February comparisons showed no matching spike.

Source evidence (1)

000051Feb 21, 2023 / 14:00 UTC-06:00

H2 is out. Pulled the historical Feb comps with the same cohort logic as the Jan scan + Owen's check this morning, and 2021 / 2022 are both basically boring: 2.0-2.2% churn, no matching spike at all. So I'm not buying competitor-cycle or some generic seasonal thing here. Owen's drafting the 1-pager now. That's two down, two left on the board: H3 onboarding-timing, H4 CS coverage. Marcus can keep waving the pricing-page experiment around if he wants, but it doesn't explain this cut.

Message 000051 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_checkpoint_annotation",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_193_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_checkpoint_annotation",
  "path": "args",
  "criterion": "The document clearly presents this as the February 20 H2 historical same-week comparison checkpoint and explicitly keeps the assessment tentative or pre-ruling, rather than describing it as a final H2 ruling.\n\nThe document accurately states that February 2022 mid-segment monthly churn was 2.2% and February 2021 mid-segment monthly churn was 2.0%.\n\nThe document states that the comparison used the same cohort logic as the January scan and Owen check.\n\nThe document explains that there was no matching historical spike and that this was directional evidence against the seasonal or competitor-cycle explanation, without elevating that evidence to a final ruling.\n\nThe document identifies one more denominator-handling sanity check as still outstanding before communicating the H2 conclusion strongly or formally.",
  "check_id": "riley_193_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_checkpoint_annotation",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_193_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_checkpoint_annotation",
        "path": "args",
        "criterion": "The document clearly presents this as the February 20 H2 historical same-week comparison checkpoint and explicitly keeps the assessment tentative or pre-ruling, rather than describing it as a final H2 ruling.\n\nThe document accurately states that February 2022 mid-segment monthly churn was 2.2% and February 2021 mid-segment monthly churn was 2.0%.\n\nThe document states that the comparison used the same cohort logic as the January scan and Owen check.\n\nThe document explains that there was no matching historical spike and that this was directional evidence against the seasonal or competitor-cycle explanation, without elevating that evidence to a final ruling.\n\nThe document identifies one more denominator-handling sanity check as still outstanding before communicating the H2 conclusion strongly or formally.",
        "check_id": "riley_193_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/193.yaml

SHA-256: b886cad7798bd6a5124f520374708621d712df8bf92e719d9c12300ea2611ed5