Test 187

Jan 1, 2028 / 1 fact

Request

Please review `doc_q1_2028_pricing_path_board_draft` and post a correction comment. Replace the draft’s unsupported causal and revenue interpretation with the certified coverage and path-quality evidence, and state the proper evidence classification clearly enough for the board editor to apply it.

Required memory

Fact 175

Riley Tanaka requested that pricing-page path quality be marked `board-ready` based on November 26, 2025 through February 28, 2026 coverage of 9,604 organic sessions: 143 of 1,981 Helio Start eligibility-flow starts, or 7.2%, were later routed out for an existing unsupported condition; eligible-workspace admission completion was 44.0%; and account-level 90th-percentile support time was 75 minutes. Because the control was retired on November 26, the row must frame the evidence as post-rollout continuity rather than a new causal or revenue claim.

Source evidence (1)

003556Mar 16, 2026 / 11:18 UTC-05:00

The March 16 source lock is complete. Update `Quarterly board operating evidence intake — Q1 2026` with the locked evidence, keeping the four lanes and their readiness statuses separate. Helio Start remains `not board-ready`: include the two certified mature cohorts and the March 3 technical-gate decision, making clear that they permit consideration of a capped proposal rather than establish acquisition economics. The exact cohort start and end dates for both sets of admitted workspaces with complete day-90 windows are still missing. Mark Expansion Assist `board-ready`: 18 actual exposures from December 1 through January 30, all with complete 45-day windows; six help-or-options requests and four expansions; no randomized control; every exposure met the standing current-state criteria; and no account had active Customer Success adoption or renewal work at exposure. Keep the claim descriptive, with no causal, revenue, or forecast treatment. Mark pricing-page path quality `board-ready`: November 26 through February 28 coverage of 9,604 organic sessions; 143 of 1,981 Helio Start eligibility-flow starts, or 7.2%, later routed out for an existing unsupported condition; 44.0% eligible-workspace admission completion; and account-level support time at the 90th percentile of 75 minutes. Because the control was retired November 26, frame this as post-rollout continuity rather than a new causal or revenue claim. Mark system controls `board-ready`: January 1 through February 28 coverage of 2,146 CS auto-nudge decisions, with 83 visible field-specific holds and 2,063 decisions that proceeded or were suppressed using current state; no governed-field disagreement, customer correction, or rollback; and two source-age alerts that fired before breach, with both affected sources refreshed before their limits. Set the outcome window to `not applicable — continuous control coverage` and limit the claim to system-control performance. For every row, preserve the lane-specific readiness model and include the applicable denominator or exact coverage dates, measurement limits, exclusions, source owner, source or release-reliability evidence, revenue and forecast treatment, and final claim owner. No unsupported revenue or forecast treatment should enter the board operating section.

Message 003556 in history

Expected tool calls

post_doc_comment

Grading

1. field_equals / post_doc_comment

{
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "pricing_path_correction",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_187_00"
}

2. field_equals / post_doc_comment

{
  "check_id": "riley_187_01",
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "pricing_path_correction",
  "path": "args.doc_id",
  "value": "doc_q1_2028_pricing_path_board_draft"
}

3. field_llm_judge / post_doc_comment

{
  "type": "field_llm_judge",
  "tool": "post_doc_comment",
  "action_id": "pricing_path_correction",
  "path": "args",
  "criterion": "The comment must accurately provide all certified pricing-page path-quality figures: coverage from November 26 through February 28; 9,604 organic sessions; 143 of 1,981 Helio Start eligibility-flow starts, or 7.2%, later routed out for an existing unsupported condition; 44.0% eligible-workspace admission completion; and 75 minutes for account-level support time at the 90th percentile. Missing or contradictory figures fail.\n\nThe comment must clearly explain that the control was retired on November 26, so the November 26–February 28 evidence is post-rollout continuity evidence, not a new or second causal test, and it must explicitly reject the draft's revenue-result interpretation. Merely hedging the causal or revenue language without correcting it fails.",
  "check_id": "riley_187_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "pricing_path_correction",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_187_00"
      },
      {
        "check_id": "riley_187_01",
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "pricing_path_correction",
        "path": "args.doc_id",
        "value": "doc_q1_2028_pricing_path_board_draft"
      },
      {
        "type": "field_llm_judge",
        "tool": "post_doc_comment",
        "action_id": "pricing_path_correction",
        "path": "args",
        "criterion": "The comment must accurately provide all certified pricing-page path-quality figures: coverage from November 26 through February 28; 9,604 organic sessions; 143 of 1,981 Helio Start eligibility-flow starts, or 7.2%, later routed out for an existing unsupported condition; 44.0% eligible-workspace admission completion; and 75 minutes for account-level support time at the 90th percentile. Missing or contradictory figures fail.\n\nThe comment must clearly explain that the control was retired on November 26, so the November 26–February 28 evidence is post-rollout continuity evidence, not a new or second causal test, and it must explicitly reject the draft's revenue-result interpretation. Merely hedging the causal or revenue language without correcting it fails.",
        "check_id": "riley_187_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/187.yaml

SHA-256: b68026347f3a42be91a3a269565c24649dfa5976333d3ddf97a06d11d2cb3b90