DolphinBench

Test 173

Jan 1, 2028 / 2 facts

YAML

Request

Create `Expansion Assist cohort comparison — Q1 vs Jul–Sep 2026` as two like-structured historical entries. Include each period, denominator, complete-window status, request and expansion counts and rates, and Customer Success exclusion. Keep cohorts separate, state whether the evidence included a randomized comparison, and explain what that means for interpreting the observed rates.

Required memory

Fact 174

Riley Tanaka requested that Expansion Assist be marked `board-ready` based on 18 actual exposures from December 1, 2025 through January 30, 2026, all with complete 45-day windows, including six help-or-options requests and four expansions. The row must state that there was no randomized control, every exposure met the standing current-state criteria, and no account had active Customer Success adoption or renewal work at exposure, and it must limit the claim to descriptive evidence with no causal, revenue, or forecast treatment.

Source evidence (1)

003556Mar 16, 2026 / 11:18 UTC-05:00

The March 16 source lock is complete. Update `Quarterly board operating evidence intake — Q1 2026` with the locked evidence, keeping the four lanes and their readiness statuses separate. Helio Start remains `not board-ready`: include the two certified mature cohorts and the March 3 technical-gate decision, making clear that they permit consideration of a capped proposal rather than establish acquisition economics. The exact cohort start and end dates for both sets of admitted workspaces with complete day-90 windows are still missing. Mark Expansion Assist `board-ready`: 18 actual exposures from December 1 through January 30, all with complete 45-day windows; six help-or-options requests and four expansions; no randomized control; every exposure met the standing current-state criteria; and no account had active Customer Success adoption or renewal work at exposure. Keep the claim descriptive, with no causal, revenue, or forecast treatment. Mark pricing-page path quality `board-ready`: November 26 through February 28 coverage of 9,604 organic sessions; 143 of 1,981 Helio Start eligibility-flow starts, or 7.2%, later routed out for an existing unsupported condition; 44.0% eligible-workspace admission completion; and account-level support time at the 90th percentile of 75 minutes. Because the control was retired November 26, frame this as post-rollout continuity rather than a new causal or revenue claim. Mark system controls `board-ready`: January 1 through February 28 coverage of 2,146 CS auto-nudge decisions, with 83 visible field-specific holds and 2,063 decisions that proceeded or were suppressed using current state; no governed-field disagreement, customer correction, or rollback; and two source-age alerts that fired before breach, with both affected sources refreshed before their limits. Set the outcome window to `not applicable — continuous control coverage` and limit the claim to system-control performance. For every row, preserve the lane-specific readiness model and include the applicable denominator or exact coverage dates, measurement limits, exclusions, source owner, source or release-reliability evidence, revenue and forecast treatment, and final claim owner. No unsupported revenue or forecast treatment should enter the board operating section.

Message 003556 in history

Fact 226

The certified Expansion Assist evidence covered 16 actual exposures from July 1 through September 15, 2026 with complete 45-day windows; six resulted in help-or-options requests and four in expansions. No exposed account had active Customer Success adoption or renewal work, and the evidence is descriptive rather than causal.

Source evidence (1)

003762Nov 6, 2026 / 15:24 UTC-06:00

Owen completed the scheduled source-closure summary. He independently normalized and reconciled the engineering decision and reliability sources through October 31, confirmed that every required field is present, established definitions are unchanged, every named owner certified their section, and no cross-surface discrepancy exists. He therefore closed the first complete four-lane source packet under his September 24 authority. The certified packet shows: - Helio Start authority-checklist evidence, September 15 through October 31: 148 newly admitted organic workspaces, 45 checklist exposures, and seven ordinary setup-assistance requests, with no eligibility-route mismatch, lifecycle-suppression bypass, paid traffic, or unsupported admission. - Expansion Assist: 16 actual exposures from July 1 through September 15 with complete 45-day windows; six help-or-options requests and four expansions. No exposed account had active Customer Success adoption or renewal work, and the evidence remains descriptive. - Pricing-page continuity, July 1 through October 31: 12,426 organic sessions and 2,576 Helio Start eligibility-flow starts; 183, or 7.1%, were later routed out for an existing unsupported condition. Eligible-workspace admission completion was 44.4%, and account-level support time at the 90th percentile was 74 minutes. - CS auto-nudge coverage, July 1 through October 31: 6,704 unique decisions, including 249 visible field-specific holds and 6,455 decisions that proceeded or were suppressed using current state. There was no governed-field disagreement, customer correction, or rollback; four source-age alerts fired and resolved before breach. I received Owen’s closure-and-exception summary without doing a line-by-line source review. The earlier coverage discrepancy is resolved and the packet is complete. Causal, revenue, forecast, board-readiness, and executive-language decisions remain open in their existing owner lanes.

Message 003762 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_comparison_sheet",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_173_00"
}
2. field_equals / create_doc
{
  "check_id": "riley_173_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_comparison_sheet",
  "path": "args.title",
  "value": "Expansion Assist cohort comparison — Q1 vs Jul–Sep 2026"
}
3. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_comparison_sheet",
  "path": "args",
  "criterion": "The document contains a distinct entry for the locked Q1 cohort that accurately states: exposure period December 1 through January 30; fixed denominator of 18 actual exposures; all exposures have complete 45-day outcome windows; 6 help-or-options requests, calculated as 33.3% of 18; 4 expansions, calculated as 22.2% of 18; and no account had active Customer Success adoption or renewal work at exposure. Missing, materially ambiguous, or contradictory values must fail.\n\nThe document contains a separate entry for the later cohort that accurately states: exposure period July 1 through September 15; fixed denominator of 16 actual exposures; all exposures have complete 45-day outcome windows; 6 help-or-options requests, calculated as 37.5% of 16; 4 expansions, calculated as 25.0% of 16; and no exposed account had active Customer Success adoption or renewal work. Missing, materially ambiguous, or contradictory values must fail.\n\nThe document keeps the two cohorts and their denominators separate rather than pooling them, and gives a shared interpretation that treats the observed request and expansion rates as descriptive only. It must not claim that the later cohort caused lift or otherwise establish causal improvement; it should make clear that the evidence does not support such a causal conclusion, including the absence of a randomized or claim-specific controlled comparison where relevant.",
  "check_id": "riley_173_02"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_comparison_sheet",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_173_00"
      },
      {
        "check_id": "riley_173_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_comparison_sheet",
        "path": "args.title",
        "value": "Expansion Assist cohort comparison — Q1 vs Jul–Sep 2026"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_comparison_sheet",
        "path": "args",
        "criterion": "The document contains a distinct entry for the locked Q1 cohort that accurately states: exposure period December 1 through January 30; fixed denominator of 18 actual exposures; all exposures have complete 45-day outcome windows; 6 help-or-options requests, calculated as 33.3% of 18; 4 expansions, calculated as 22.2% of 18; and no account had active Customer Success adoption or renewal work at exposure. Missing, materially ambiguous, or contradictory values must fail.\n\nThe document contains a separate entry for the later cohort that accurately states: exposure period July 1 through September 15; fixed denominator of 16 actual exposures; all exposures have complete 45-day outcome windows; 6 help-or-options requests, calculated as 37.5% of 16; 4 expansions, calculated as 25.0% of 16; and no exposed account had active Customer Success adoption or renewal work. Missing, materially ambiguous, or contradictory values must fail.\n\nThe document keeps the two cohorts and their denominators separate rather than pooling them, and gives a shared interpretation that treats the observed request and expansion rates as descriptive only. It must not claim that the later cohort caused lift or otherwise establish causal improvement; it should make clear that the evidence does not support such a causal conclusion, including the absence of a randomized or claim-specific controlled comparison where relevant.",
        "check_id": "riley_173_02"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/173.yaml

SHA-256: 92f971a67cc0e7010c79a4539ffb0d7a94eb0e08ae7cd716d48ba1a93c298b9f