Test 071

Jan 1, 2028 / 1 fact

Request

Create a compact Helio Start public-intake routing reference using the rules approved for the March 2024 launch.

Required memory

Fact 13

Helio Start’s public self-serve path is limited to under-50 accounts with admin proof, simple deployment, first-week integration fit, a current entitlement state, and supported billing conditions. Legitimate domain mismatches may use corroborating evidence, while agency identities, manual billing, imports, migrations, and unsupported permissions must route out.

Source evidence (1)

001211Mar 4, 2024 / 15:05 UTC-06:00

The Helio Start decision meeting just ended. Tomás approved a bounded March 18 general-availability launch on the public pricing page, but paid acquisition and any forecastable pipeline language are still out. The public path is only for under-50 accounts when admin proof, deployment simplicity, first-week integration fit, a current entitlement state, and supported billing conditions are all present. Legitimate domain mismatches can use corroborating evidence, while agency identities, manual billing, imports, migrations, and unsupported permissions route out instead of entering self-serve. Priya also expanded my remit to the full self-serve operating model across eligibility and product constraints, activation and support-load measurement, lifecycle, support boundaries, and executive language, without creating a new team. Support load stays a first-class metric beside activation. Owen now owns the reporting layer and has to keep Helio Start activation and support-load measurement separate from retained mid-segment and Expansion Assist. Daniela owns entitlement and support-state instrumentation, Ines owns lifecycle copy and risk controls, and Customer Success owns routed-before-setup support paths.

Message 001211 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_routing_reference",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_071_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_routing_reference",
  "path": "args",
  "criterion": "The reference says the public self-serve path accepts under-50 accounts only when they have admin proof, simple deployment, first-week integration fit, current entitlement, and supported billing. A legitimate domain mismatch may use corroborating evidence. Agency identities, manual billing, imports, migrations, and unsupported permissions route out.",
  "check_id": "riley_071_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_routing_reference",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_071_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_routing_reference",
        "path": "args",
        "criterion": "The reference says the public self-serve path accepts under-50 accounts only when they have admin proof, simple deployment, first-week integration fit, current entitlement, and supported billing. A legitimate domain mismatch may use corroborating evidence. Agency identities, manual billing, imports, migrations, and unsupported permissions route out.",
        "check_id": "riley_071_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/071.yaml

SHA-256: d8948bed3b653402f176a63536eabd2b789e77c058f5fdec16396c5e92b21747