Test 184

Jan 1, 2028 / 1 fact

Request

Please review the support-effort claim in `doc_helio_start_2028_authority_checklist_proposal` and post a correction comment that replaces the causal improvement framing with the certified mature authority-checklist cohort evidence and its appropriate limitation.

Required memory

Fact 247

Owen and Customer Success completed and certified the 90-day account-level support cut for all 45 organic Helio Start workspaces that received the authority checklist from September 15 through October 31. All windows are complete; median support effort was 24 minutes, the 90th percentile was 86 minutes, three workspaces exceeded 90 minutes, and no unsupported case entered the admitted path. The result is descriptive because there was no randomized control.

Source evidence (1)

004114Mar 17, 2027 / 13:10 UTC-05:00

Owen and Customer Success have completed and certified the 90-day account-level support cut for all 45 organic Helio Start workspaces that received the authority checklist from September 15 through October 31. All 45 have complete windows. Support effort was 24 minutes at the median and 86 minutes at the 90th percentile; three of 45 exceeded 90 support minutes, and no unsupported case entered the admitted path. The result remains descriptive because there was no randomized control. Customer Success separately certified that it could staff a future capped run of no more than 25 admitted workspaces without displacing ordinary assisted-path coverage. Priya set a March 24 decision on whether these inputs justify developing, but not launching, a Q2 proposal. Replace the preliminary section in `Lifecycle Systems Q2 kickoff — March 18 open questions` with the certified input, then post a concise summary in `#lifecycle-systems` preserving the descriptive limitation, 25-admission cap, March 24 decision boundary, and the fact that no proposal development, launch, budget, paid traffic, pipeline treatment, forecast credit, eligibility change, lifecycle bypass, or fifth shared-state consumer is authorized now.

Message 004114 in history

Expected tool calls

post_doc_comment

Grading

1. field_equals / post_doc_comment

{
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_authority_checklist_correction",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_184_00"
}

2. field_equals / post_doc_comment

{
  "check_id": "riley_184_01",
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_authority_checklist_correction",
  "path": "args.doc_id",
  "value": "doc_helio_start_2028_authority_checklist_proposal"
}

3. field_llm_judge / post_doc_comment

{
  "type": "field_llm_judge",
  "tool": "post_doc_comment",
  "action_id": "post_authority_checklist_correction",
  "path": "args",
  "criterion": "The comment identifies the evidence as covering all 45 organic Helio Start workspaces that received the authority checklist from September 15 through October 31, and states that all 45 have complete 90-day account-level support windows. Missing or contradictory cohort, date-window, denominator, or completeness information must fail.\n\nThe comment reports support effort of 24 minutes at the median and 86 minutes at the 90th percentile, and states that 3 of 45 workspaces exceeded 90 support minutes. Missing or contradictory values must fail.\n\nThe comment states that no unsupported case entered the admitted path. Missing or contradictory language must fail.\n\nThe comment explicitly corrects or rejects the proposal's causal support-improvement framing and says the cohort result is descriptive, not causal, because there was no randomized control. Generic caution without both the descriptive limitation and the lack of randomized control must fail.",
  "check_id": "riley_184_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_authority_checklist_correction",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_184_00"
      },
      {
        "check_id": "riley_184_01",
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_authority_checklist_correction",
        "path": "args.doc_id",
        "value": "doc_helio_start_2028_authority_checklist_proposal"
      },
      {
        "type": "field_llm_judge",
        "tool": "post_doc_comment",
        "action_id": "post_authority_checklist_correction",
        "path": "args",
        "criterion": "The comment identifies the evidence as covering all 45 organic Helio Start workspaces that received the authority checklist from September 15 through October 31, and states that all 45 have complete 90-day account-level support windows. Missing or contradictory cohort, date-window, denominator, or completeness information must fail.\n\nThe comment reports support effort of 24 minutes at the median and 86 minutes at the 90th percentile, and states that 3 of 45 workspaces exceeded 90 support minutes. Missing or contradictory values must fail.\n\nThe comment states that no unsupported case entered the admitted path. Missing or contradictory language must fail.\n\nThe comment explicitly corrects or rejects the proposal's causal support-improvement framing and says the cohort result is descriptive, not causal, because there was no randomized control. Generic caution without both the descriptive limitation and the lack of randomized control must fail.",
        "check_id": "riley_184_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/184.yaml

SHA-256: 6672aba744ea9982185a08ac0287011a9e8261d0ca5efdfcc384df56bec5c86e