Test 135 for Riley Tanaka

Request

Please create a compact historical adjudication worksheet for the controlled-intake archive to train reviewers on identity and state handling. Use a decision-matrix format with sample rows for exact-domain proof, a legitimate parent-company or IT-managed domain mismatch, an agency identity, and missing current-state data. Include the routing or adjudication result for each row and the final entitlement and support-state validation checks. Clearly label the worksheet’s scope so reviewers do not treat this historical controlled-intake matrix as universal guidance for every later Helio Start flow.

Required memory

Fact 4

Daniela and Owen completed a beta-only evidence-tier dry run on 30 held January Helio Start applicants: 18 exact-domain cases passed automatically, nine legitimate parent-company or IT-managed identities passed only after corroborating admin evidence, and three agency identities were routed to Customer Success instead of self-attestation.

Source evidence (1)

001035Jan 9, 2024 / 14:30 UTC-06:00

Daniela and Owen finished the beta-only evidence-tier dry run on 30 held January Helio Start applicants. Eighteen exact-domain cases passed automatically, nine legitimate parent-company or IT-managed identities only passed once they had corroborating admin evidence, and three agency identities were routed to Customer Success instead of self-attestation. Every applicant row produced exactly one current entitlement state and one support state, with no blanks, duplicate workspace records, or merged states. I'm accepting that as the intake rule for the next controlled batch, but I'm not broadening exposure from here. Please create a January 18, 2024 calendar event from 2:00pm to 2:45pm Central for me, Priya, Daniela, Owen, Ines, and Customer Success. The notes should cover support load, admin proof, entitlement and support-state integrity, and the unsupported lifecycle list, and they need to say explicitly that this is a controlled-beta readiness gate rather than a GA, public signup, pricing-page, or paid-acquisition decision.

Message 001035 in history

Fact 5

The dry run produced exactly one current entitlement state and one support state for every applicant row, with no blanks, duplicate workspace records, or merged states.

Source evidence (1)

001035Jan 9, 2024 / 14:30 UTC-06:00

Daniela and Owen finished the beta-only evidence-tier dry run on 30 held January Helio Start applicants. Eighteen exact-domain cases passed automatically, nine legitimate parent-company or IT-managed identities only passed once they had corroborating admin evidence, and three agency identities were routed to Customer Success instead of self-attestation. Every applicant row produced exactly one current entitlement state and one support state, with no blanks, duplicate workspace records, or merged states. I'm accepting that as the intake rule for the next controlled batch, but I'm not broadening exposure from here. Please create a January 18, 2024 calendar event from 2:00pm to 2:45pm Central for me, Priya, Daniela, Owen, Ines, and Customer Success. The notes should cover support load, admin proof, entitlement and support-state integrity, and the unsupported lifecycle list, and they need to say explicitly that this is a controlled-beta readiness gate rather than a GA, public signup, pricing-page, or paid-acquisition decision.

Message 001035 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_intake_matrix",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_135_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_intake_matrix",
  "path": "args",
  "criterion": "The document contains a clearly identifiable decision-matrix sample row for exact-domain admin proof and states that this identity-evidence tier passes automatically. The row may and should still remain subject to final entitlement-state, support-state, and any other controlled-intake validations before completed admission; those downstream checks do not negate automatic passage of the identity-evidence tier.\n\nThe document contains a clearly identifiable decision-matrix sample row for a legitimate parent-company or IT-managed domain mismatch, and it states that the applicant may pass only after corroborating admin evidence is provided—not through unsupported self-attestation.\n\nThe document contains a clearly identifiable decision-matrix sample row for an agency identity and routes that case to Customer Success rather than allowing it to pass through self-attestation.\n\nThe document includes a sample row with missing current-state data that cannot receive a completed or passing final adjudication until corrected, and it gives final validation checks requiring exactly one current entitlement state and exactly one current support state for every applicant, with blank, duplicate, or merged state results not accepted.\n\nThe document explicitly identifies the matrix as historical guidance for the Helio Start controlled-intake process or controlled batch and warns that it is not universal guidance for every later Helio Start flow.",
  "check_id": "riley_135_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_intake_matrix",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_135_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_intake_matrix",
        "path": "args",
        "criterion": "The document contains a clearly identifiable decision-matrix sample row for exact-domain admin proof and states that this identity-evidence tier passes automatically. The row may and should still remain subject to final entitlement-state, support-state, and any other controlled-intake validations before completed admission; those downstream checks do not negate automatic passage of the identity-evidence tier.\n\nThe document contains a clearly identifiable decision-matrix sample row for a legitimate parent-company or IT-managed domain mismatch, and it states that the applicant may pass only after corroborating admin evidence is provided—not through unsupported self-attestation.\n\nThe document contains a clearly identifiable decision-matrix sample row for an agency identity and routes that case to Customer Success rather than allowing it to pass through self-attestation.\n\nThe document includes a sample row with missing current-state data that cannot receive a completed or passing final adjudication until corrected, and it gives final validation checks requiring exactly one current entitlement state and exactly one current support state for every applicant, with blank, duplicate, or merged state results not accepted.\n\nThe document explicitly identifies the matrix as historical guidance for the Helio Start controlled-intake process or controlled batch and warns that it is not universal guidance for every later Helio Start flow.",
        "check_id": "riley_135_01"
      }
    ]
  }
}