Test 072

Jan 1, 2028 / 1 fact

Request

Create a compact dated benchmark card for the shared state feed’s first three live weeks, using the ledger close date. Include the evaluated-record count, the stale-or-conflicting fail-closed count, the calculated fail-closed share, and whether eligibility, reporting, and messaging disagreed on authority, active-admin, entitlement, or Customer Success-work status. Save it as a new document.

Required memory

Fact 88

Across two Helio Start reporting cycles, the March Expansion Assist selection, and three lifecycle preflights, the shared state feed evaluated 638 records; 22 stale or conflicting records failed closed, and no record had differing authority, active-admin, entitlement, or Customer Success-work status across eligibility, reporting, and messaging.

Source evidence (1)

002771Mar 28, 2025 / 09:15 UTC-05:00

Owen closed the three-week live ledger after the final evaluated records arrived. Across two Helio Start reporting cycles, the March Expansion Assist selection, and three lifecycle preflights, the shared state feed evaluated 638 records. Twenty-two stale or conflicting records failed closed, and no record carried a different authority, active-admin, entitlement, or Customer Success-work status between eligibility, reporting, and messaging. Owen owns and delivers the exception readout without me rechecking every row. Priya reviewed the result and made the shared state contract mandatory for new Growth and Customer Success lifecycle decisioning. Daniela scheduled the CS auto-nudge system as the next shared-state integration in Q2; the nudge is not yet integrated, and the result does not make Lifecycle Systems companywide or mature.

Message 002771 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "benchmark_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_072_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "benchmark_doc",
  "path": "args",
  "criterion": "The title or body clearly identifies the document as a benchmark card or benchmark for the shared state feed's first live period.\n\nThe title or body clearly dates the benchmark to the ledger close date of March 28, 2025.\n\nThe body reports that 638 records were evaluated and that 22 stale or conflicting records failed closed (the held/fail-closed count). Missing either count or attributing a different count must fail.\n\nThe body gives a numeric fail-closed share correctly calculated from 22 divided by 638, such as approximately 3.45% or an equivalently correct rounded percentage.\n\nThe body states that there was no disagreement across eligibility, reporting, and messaging for the governed authority, active-admin, entitlement, or Customer Success-work statuses. Any claim of a disagreement, or omission of the no-disagreement result, must fail.",
  "check_id": "riley_072_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "benchmark_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_072_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "benchmark_doc",
        "path": "args",
        "criterion": "The title or body clearly identifies the document as a benchmark card or benchmark for the shared state feed's first live period.\n\nThe title or body clearly dates the benchmark to the ledger close date of March 28, 2025.\n\nThe body reports that 638 records were evaluated and that 22 stale or conflicting records failed closed (the held/fail-closed count). Missing either count or attributing a different count must fail.\n\nThe body gives a numeric fail-closed share correctly calculated from 22 divided by 638, such as approximately 3.45% or an equivalently correct rounded percentage.\n\nThe body states that there was no disagreement across eligibility, reporting, and messaging for the governed authority, active-admin, entitlement, or Customer Success-work statuses. Any claim of a disagreement, or omission of the no-disagreement result, must fail.",
        "check_id": "riley_072_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/072.yaml

SHA-256: 062aa3f7e8d947e08e7a24456985f6c6a9d2c59a7f76647fe4bd1cfc49ffee62