Test 077

Jan 1, 2028 / 3 facts

Request

Create a compact document using this exact title: `H2 evidence-quality annotation — Feb. 20 checkpoint`. Treat it as a historical snapshot of directional evidence before a formal ruling. Record the same-window comparison, continuity with the earlier analysis method, what direction the evidence pointed, and the unresolved quality check at that time.

Required memory

Fact 321

On February 20, 2023, Riley Tanaka used the same cohort logic as the January scan and Owen check for an H2 same-week comparison: mid-segment monthly churn was 2.2% in February 2022 and 2.0% in February 2021, showing no apparent historical counterpart to the current spike.

Source evidence (1)

000049Feb 20, 2023 / 11:00 UTC-06:00

pulled the same-week comp for h2. Feb 2022 mid-seg monthly churn is 2.2%. Feb 2021 is 2.0%. so unless i've broken the cut in some new fun way, there just isn't a historical version of this spike sitting there. using the same cohort logic as the Jan scan + the Owen check this morning. h1 already looked dead after the tier split came back flat, and this pushes pretty hard against the easy "seasonal" / competitor-cycle story too. still want to sanity check denominator handling once more before i say that too loudly -- but directionally this is bad for the comforting explanations. Marcus is still trying to run the pricing-page thing in parallel lol

Message 000049 in history

Fact 323

On February 21, 2023, Riley Tanaka ruled out H2—competitor cycle or a generic seasonal effect—as an explanation for the mid-segment churn spike because historical February comparisons showed no matching spike.

Source evidence (1)

000051Feb 21, 2023 / 14:00 UTC-06:00

H2 is out. Pulled the historical Feb comps with the same cohort logic as the Jan scan + Owen's check this morning, and 2021 / 2022 are both basically boring: 2.0-2.2% churn, no matching spike at all. So I'm not buying competitor-cycle or some generic seasonal thing here. Owen's drafting the 1-pager now. That's two down, two left on the board: H3 onboarding-timing, H4 CS coverage. Marcus can keep waving the pricing-page experiment around if he wants, but it doesn't explain this cut.

Message 000051 in history

Fact 322

Riley Tanaka considers the historical comparison directional evidence against a seasonal or competitor-cycle explanation, but wants one more denominator-handling sanity check before stating that conclusion strongly.

Source evidence (1)

000049Feb 20, 2023 / 11:00 UTC-06:00

Message 000049 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_h2_annotation",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_077_00"
}

2. field_equals / create_doc

{
  "check_id": "riley_077_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_h2_annotation",
  "path": "args.title",
  "value": "H2 evidence-quality annotation — Feb. 20 checkpoint"
}

3. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_h2_annotation",
  "path": "args",
  "criterion": "The body accurately records the historical same-window comparison: February 2022 mid-segment monthly churn was 2.2%, February 2021 was 2.0%, and there was no comparable historical spike. Missing or contradicting either year, value, or the absence of a matching spike must fail.\n\nThe body states that the historical comparison used the same cohort logic as the January scan and Owen check. A generic claim that the method was consistent, without identifying those prior checks, is insufficient.\n\nThe body frames the Feb. 20 evidence as directional evidence against the seasonal and/or competitor-cycle explanation, while clearly stating that an additional denominator-handling sanity check was still outstanding and therefore the conclusion should not yet be stated as a formal ruling at that checkpoint. It must preserve this as a dated historical pre-ruling snapshot rather than imply the check remains unresolved today.",
  "check_id": "riley_077_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_h2_annotation",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_077_00"
      },
      {
        "check_id": "riley_077_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_h2_annotation",
        "path": "args.title",
        "value": "H2 evidence-quality annotation — Feb. 20 checkpoint"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_h2_annotation",
        "path": "args",
        "criterion": "The body accurately records the historical same-window comparison: February 2022 mid-segment monthly churn was 2.2%, February 2021 was 2.0%, and there was no comparable historical spike. Missing or contradicting either year, value, or the absence of a matching spike must fail.\n\nThe body states that the historical comparison used the same cohort logic as the January scan and Owen check. A generic claim that the method was consistent, without identifying those prior checks, is insufficient.\n\nThe body frames the Feb. 20 evidence as directional evidence against the seasonal and/or competitor-cycle explanation, while clearly stating that an additional denominator-handling sanity check was still outstanding and therefore the conclusion should not yet be stated as a formal ruling at that checkpoint. It must preserve this as a dated historical pre-ruling snapshot rather than imply the check remains unresolved today.",
        "check_id": "riley_077_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/077.yaml

SHA-256: 62697d401886724787b3d07cea8d62d2fa21242f26be2b26402f646b1bf68894