Test 011

Jan 1, 2028 / 2 facts

Request

Create a Helio Start archive note for onboarding and retrospective use that captures the original sub-50-seat effort’s duration, core workstreams, and how commercial guardrails were meant to relate to eligibility and product constraints.

Required memory

Fact 388

Riley Tanaka scoped 12 weeks to establish the sub-50-seat tier’s foundation through a funnel event taxonomy redo, pricing page redesign, lifecycle program changes, and dashboard build-out.

Source evidence (1)

000264May 26, 2023 / 10:00 UTC-05:00

meeting done. I had the cleanest read in the room tbh. net is yes on a sub-50-seat tier, but not as a fast toggle -- I scoped 12 weeks just to get the bones in place: funnel event taxonomy redo, pricing page redesign, lifecycle program changes, dashboard build-out. daniela was basically yes-if-we-stop pretending eng can absorb this sideways. tomas green-lit pre-discovery for june and put me on point for it. marcus/priya seemed aligned once the instrumentation piece was explicit.

Message 000264 in history

Fact 383

Riley Tanaka is splitting the small-business-tier document into packaging/pricing guardrails and eligibility/product constraints because a cheaper plan does not resolve seat-floor, SSO, support-load, and onboarding-touch issues.

Source evidence (1)

000253May 20, 2023 / 15:21 UTC-05:00

small-business-tier doc is basically two docs jammed together rn. one is packaging/pricing guardrails, other is actual eligibility + product constraints. i'm splitting those -- otherwise we keep pretending a cheap plan fixes the stuff that is really seat floor, SSO, support load, and onboarding touch. leaning to define SMB first by deployment shape + admin complexity, not just logo size. if a 35-seat account needs SCIM and legal redlines, that's not the same problem as a 90-seat self-serve-ish team. also putting in a hard note that expansion math has to exclude the retained mid-seg cohort work -- different motion, can't let Marcus blend them.

Message 000253 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_helio_start_archive",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_011_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_helio_start_archive",
  "path": "args",
  "criterion": "The document body clearly records the original sub-50-seat effort as a 12-week effort and includes all four core workstreams: funnel event taxonomy redo, pricing page redesign, lifecycle program changes, and dashboard build-out. Equivalent wording is acceptable, but all four workstreams and the 12-week duration must be present.\n\nThe document body explains that the scoping was intended to keep packaging/pricing guardrails separate from eligibility and product constraints rather than combining them into one undifferentiated section. Equivalent wording is acceptable.",
  "check_id": "riley_011_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_helio_start_archive",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_011_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_helio_start_archive",
        "path": "args",
        "criterion": "The document body clearly records the original sub-50-seat effort as a 12-week effort and includes all four core workstreams: funnel event taxonomy redo, pricing page redesign, lifecycle program changes, and dashboard build-out. Equivalent wording is acceptable, but all four workstreams and the 12-week duration must be present.\n\nThe document body explains that the scoping was intended to keep packaging/pricing guardrails separate from eligibility and product constraints rather than combining them into one undifferentiated section. Equivalent wording is acceptable.",
        "check_id": "riley_011_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/011.yaml

SHA-256: db4f00d9d6788ee77cc160ac695387001102c0edad75366f240ed5a168ce5bc7