Test 186

Jan 1, 2028 / 3 facts

Request

Post a short January reminder in #helio-start-ops summarizing the current public self-serve applicant guardrails. Distinguish applicants routed to assistance from applicants determined ineligible, and state what happens to unresolved cases.

Required memory

Fact 13

Helio Start’s public self-serve path is limited to under-50 accounts with admin proof, simple deployment, first-week integration fit, a current entitlement state, and supported billing conditions. Legitimate domain mismatches may use corroborating evidence, while agency identities, manual billing, imports, migrations, and unsupported permissions must route out.

Source evidence (1)

001211Mar 4, 2024 / 15:05 UTC-06:00

The Helio Start decision meeting just ended. Tomás approved a bounded March 18 general-availability launch on the public pricing page, but paid acquisition and any forecastable pipeline language are still out. The public path is only for under-50 accounts when admin proof, deployment simplicity, first-week integration fit, a current entitlement state, and supported billing conditions are all present. Legitimate domain mismatches can use corroborating evidence, while agency identities, manual billing, imports, migrations, and unsupported permissions route out instead of entering self-serve. Priya also expanded my remit to the full self-serve operating model across eligibility and product constraints, activation and support-load measurement, lifecycle, support boundaries, and executive language, without creating a new team. Support load stays a first-class metric beside activation. Owen now owns the reporting layer and has to keep Helio Start activation and support-load measurement separate from retained mid-segment and Expansion Assist. Daniela owns entitlement and support-state instrumentation, Ines owns lifecycle copy and risk controls, and Customer Success owns routed-before-setup support paths.

Message 001211 in history

Fact 17

Helio Start went live on the public pricing page on March 18, 2024.

Source evidence (1)

001256Mar 18, 2024 / 09:14 UTC-05:00

Helio Start is live on the public pricing page. The live smoke test admitted the simple under-50 workspace with verified admin access, supported billing, a current entitlement state, and a first-week integration fit. The manual-billing case routed to assisted, the agency case was ineligible for Start, and the unresolved-ownership case stayed pending instead of entering a workspace. Owen's live view keeps Start activation and support load separate from ordinary starter and retained mid-segment, Ines's Start lifecycle fork is active, and no unrelated pricing-page layout change shipped with the release.

Message 001256 in history

Fact 18

The Helio Start live smoke test admitted an eligible simple under-50 workspace, routed a manual-billing case to assisted service, marked an agency case ineligible, and kept an unresolved-ownership case pending without creating a workspace.

Source evidence (1)

001256Mar 18, 2024 / 09:14 UTC-05:00

Message 001256 in history

Expected tool calls

send_slack_message

Grading

1. field_equals / send_slack_message

{
  "type": "field_equals",
  "tool": "send_slack_message",
  "action_id": "post_helio_january_reminder",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_186_00"
}

2. field_equals / send_slack_message

{
  "check_id": "riley_186_01",
  "type": "field_equals",
  "tool": "send_slack_message",
  "action_id": "post_helio_january_reminder",
  "path": "args.channel",
  "value": "#helio-start-ops"
}

3. field_llm_judge / send_slack_message

{
  "type": "field_llm_judge",
  "tool": "send_slack_message",
  "action_id": "post_helio_january_reminder",
  "path": "args",
  "criterion": "The message concisely communicates that Helio Start's live public or pricing-page self-serve path is bounded by supported applicant criteria; it must not portray the product as unrestricted self-serve or suggest that eligibility has been broadened.\n\nThe message explicitly says that applicants outside Helio Start's supported boundaries are routed to an assisted path or determined ineligible rather than being admitted into self-serve. Missing either the assisted outcome or the ineligible outcome, or implying silent admission, must fail.",
  "check_id": "riley_186_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_slack_message",
        "action_id": "post_helio_january_reminder",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_186_00"
      },
      {
        "check_id": "riley_186_01",
        "type": "field_equals",
        "tool": "send_slack_message",
        "action_id": "post_helio_january_reminder",
        "path": "args.channel",
        "value": "#helio-start-ops"
      },
      {
        "type": "field_llm_judge",
        "tool": "send_slack_message",
        "action_id": "post_helio_january_reminder",
        "path": "args",
        "criterion": "The message concisely communicates that Helio Start's live public or pricing-page self-serve path is bounded by supported applicant criteria; it must not portray the product as unrestricted self-serve or suggest that eligibility has been broadened.\n\nThe message explicitly says that applicants outside Helio Start's supported boundaries are routed to an assisted path or determined ineligible rather than being admitted into self-serve. Missing either the assisted outcome or the ineligible outcome, or implying silent admission, must fail.",
        "check_id": "riley_186_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/186.yaml

SHA-256: 933b3a88b9db7ab773c8c1e9cd1cebc60fa93ea6b328bd2c8bbfc6e07e35cfd9