DolphinBench
DiscordGitHub

Test 014

Jan 1, 2028 / 1 fact

YAML

Request

Create a reusable pre-design intake checklist for new Growth or Customer Success lifecycle decisioning. State the prerequisite and say plainly whether design may proceed before that prerequisite is satisfied.

Required memory

Fact 89

Priya reviewed the three-week live ledger result and made the shared state contract mandatory for new Growth and Customer Success lifecycle decisioning.

Source evidence (1)

002771Mar 28, 2025 / 09:15 UTC-05:00

Owen closed the three-week live ledger after the final evaluated records arrived. Across two Helio Start reporting cycles, the March Expansion Assist selection, and three lifecycle preflights, the shared state feed evaluated 638 records. Twenty-two stale or conflicting records failed closed, and no record carried a different authority, active-admin, entitlement, or Customer Success-work status between eligibility, reporting, and messaging. Owen owns and delivers the exception readout without me rechecking every row. Priya reviewed the result and made the shared state contract mandatory for new Growth and Customer Success lifecycle decisioning. Daniela scheduled the CS auto-nudge system as the next shared-state integration in Q2; the nudge is not yet integrated, and the result does not make Lifecycle Systems companywide or mature.

Message 002771 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_lifecycle_intake_checklist",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_014_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_lifecycle_intake_checklist",
  "path": "args",
  "criterion": "The created document is a reusable pre-design intake checklist scoped to new Growth or Customer Success lifecycle decisioning. It must include an unmistakably mandatory, blocking prerequisite that the new decisioning use the shared state contract before design proceeds. Language making state-contract use optional, advisory, unresolved while design advances, or only a post-design validation fails.",
  "check_id": "riley_014_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_lifecycle_intake_checklist",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_014_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_lifecycle_intake_checklist",
        "path": "args",
        "criterion": "The created document is a reusable pre-design intake checklist scoped to new Growth or Customer Success lifecycle decisioning. It must include an unmistakably mandatory, blocking prerequisite that the new decisioning use the shared state contract before design proceeds. Language making state-contract use optional, advisory, unresolved while design advances, or only a post-design validation fails.",
        "check_id": "riley_014_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/014.yaml

SHA-256: 8ec18ec67723be37227bfe6c6800e64be53a5f66f2f7dfab8ff3f02abb38a682