DolphinBench

Test 197

Jan 1, 2028 / 3 facts

YAML

Request

Create a document titled `January 2028 self-serve operating-model RACI` in the Growth folder. Keep it concise and practical for the review. Name the overall accountable owner rather than giving only a role title, summarize every covered decision area including executive language, and make clear how the two core operating outcomes should be weighted.

Required memory

Fact 79

Riley Tanaka’s approved Lifecycle Systems lane covers high-stakes cuts and Helio’s self-serve operating model, with support load measured beside activation.

Source evidence (2)

002453Jan 24, 2025 / 15:40 UTC-06:00

Record the approved January 24 decision on the existing charter proposal, announce in #growth that Lifecycle Systems begins as a formal Helio function on Monday, January 27 with me as lead, and create the January 27 operating kickoff from 10:00am to 10:45am with Priya Devarajan, Owen Pham, Ines Kovac, Daniela Schultz, and me. Include the agreed lanes: Owen on measurement, definitions, and exception reporting; Ines on lifecycle governance and message-state controls; Daniela's team as the engineering partner for event and state reliability; and me on high-stakes cuts and the self-serve operating model, including support load beside activation. Keep clear that the shared state contract is not live in production.

Message 002453 in history
004079Mar 5, 2027 / 08:05 UTC-06:00

Send a text-only email to the electricity supplier with subject `Unexpected $9.95 plan service fee on Cherrywood electricity statement` and this exact body: `Hello, my latest statement includes a separate $9.95 plan service fee. My active fixed plan runs through August 12, 2027 and has no monthly plan fee. The statement otherwise shows 721 kWh at the contracted $0.138 per kWh rate. Please identify the contractual basis for this fee or correct the statement, and confirm the result in writing. Thank you, Riley Tanaka.`

Message 004079 in history

Fact 423

Riley Tanaka owns the January Helio Start operating model across eligibility, measurement, lifecycle, support load, and executive language; this responsibility does not create a formal new team.

Source evidence (1)

000972Dec 22, 2023 / 14:35 UTC-06:00

We finished the Helio Start Q4 closeout. The accepted conclusion is intentionally narrow: simple under-50 accounts with verified admin ownership, no manual billing need, no import or migration need, and a first-week integration path can activate without recreating the August-level support mess. In the controlled October and November batches, 50 invitees were accepted; 34 had confirmed admin ownership and 16 were routed to Customer Success before setup. That produced 22 new workspaces and 19 first integrations. The group agreed that routed-before-setup is doing useful work, and the absence of admitted manual-billing or import tickets supports keeping those needs outside the beta. The evidence is enough for a January readiness review with explicit criteria, not for a public product decision. Tomás explicitly did not approve GA, GA planning, pricing-page exposure, paid acquisition, broader beta exposure, or any public launch commitment. Marcus also accepted that there is no Q4 launch forecast, forecastable pipeline, or revenue-channel language to claim. Broader exposure stays blocked until the permissions and admin path stops creating avoidable support handoffs. I own the January operating model across eligibility, measurement, lifecycle, support load, and executive language, but this is still an operating-model role rather than a formal new team. Owen owns the separate beta dashboard plus activation and support-load readout, Daniela owns entitlement and support-state instrumentation, Ines owns lifecycle copy and risk boundaries, and Customer Success owns the routed-before-setup support paths.

Message 000972 in history

Fact 412

The first-integration beta will treat support load as a first-class metric alongside activation.

Source evidence (1)

000538Aug 22, 2023 / 11:28 UTC-05:00

The late-morning beta-read discussion landed. Marcus Vail asked whether the first-integration shape supports widening the beta next week, especially because the simple-deployment accounts are moving inside the expected early window, and I said no: the permissions, imports, and billing-expectation questions are the point of this beta read, not noise to ignore. Daniela agreed the instrumentation is good enough to track support load beside activation, Owen will keep support questions per active beta account visible next to first-integration progress, and Ines tightened the beta education copy so it now says the beta supports simple first integration with docs-first help, does not support migration/import help, manual billing exceptions, or unclear admin ownership inside the self-serve path, and confirms billing expectations before anything changes. Draft a concise follow-up note for Marcus, Tomás, Priya, Owen, Ines, and Daniela that records the decision that the beta stays capped, support load is a first-class metric beside activation, and permissions, imports, and billing need to be resolved before broader exposure, without sounding anti-revenue.

Message 000538 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_self_serve_raci",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_197_00"
}
2. field_equals / create_doc
{
  "check_id": "riley_197_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_self_serve_raci",
  "path": "args.title",
  "value": "January 2028 self-serve operating-model RACI"
}
3. field_equals / create_doc
{
  "check_id": "riley_197_02",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_self_serve_raci",
  "path": "args.folder",
  "value": "Growth"
}
4. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_self_serve_raci",
  "path": "args",
  "criterion": "The document clearly names Riley as the overall accountable owner of Helio's self-serve operating model. It should frame this as overall accountability and must not assign the overall accountability to someone else.\n\nThe document concisely covers all of these self-serve operating-model decision areas: eligibility and product constraints; activation and support-load measurement; lifecycle; support boundaries; and executive language. Synonymous wording is acceptable, but none of these areas may be omitted or assigned outside the overall model.\n\nThe document identifies activation and support load as the two core operating outcomes and explicitly treats support load as a first-class peer metric beside activation, not as a secondary consideration or footnote.",
  "check_id": "riley_197_03"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_self_serve_raci",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_197_00"
      },
      {
        "check_id": "riley_197_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_self_serve_raci",
        "path": "args.title",
        "value": "January 2028 self-serve operating-model RACI"
      },
      {
        "check_id": "riley_197_02",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_self_serve_raci",
        "path": "args.folder",
        "value": "Growth"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_self_serve_raci",
        "path": "args",
        "criterion": "The document clearly names Riley as the overall accountable owner of Helio's self-serve operating model. It should frame this as overall accountability and must not assign the overall accountability to someone else.\n\nThe document concisely covers all of these self-serve operating-model decision areas: eligibility and product constraints; activation and support-load measurement; lifecycle; support boundaries; and executive language. Synonymous wording is acceptable, but none of these areas may be omitted or assigned outside the overall model.\n\nThe document identifies activation and support load as the two core operating outcomes and explicitly treats support load as a first-class peer metric beside activation, not as a secondary consideration or footnote.",
        "check_id": "riley_197_03"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/197.yaml

SHA-256: 7e7301a6a3f60b73f896705ef26aa0d0b128b7b91da0ceea33213b3efb8b638e