DolphinBench

Test 057

Jan 1, 2028 / 5 facts

YAML

Request

Create a concise Helio Start decision evidence card for the March 2024 public-product decision. Summarize the readiness cohort evidence across operating volume, support coverage, state integrity, excluded operational cases, and the limits on what the evidence justified. Make the distinction between what the cohort established and what it did not establish explicit. Create it as a new document in the Growth folder.

Required memory

Fact 8

Of the 120 Helio Start applications, 76 auto-passed with exact-match admin proof, 28 legitimate mismatches passed with corroborating admin evidence, and 16 were routed out because of agency identity, unresolved ownership, manual billing, import, or migration needs.

Source evidence (1)

001189Feb 26, 2024 / 15:08 UTC-06:00

We closed the Helio Start results meeting today. The 120 applications broke out as 76 auto-passed exact-match admin proofs, 28 legitimate mismatches passed with corroborating admin evidence, and 16 routed out for agency identity, unresolved ownership, manual billing, import, or migration needs. Of the 104 admitted accounts, 82 created workspaces and 68 reached first integration by day seven. Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage, no manual-billing, import, or migration case entered the admitted path, and Daniela's audit still showed one current entitlement state and one current support state for every admitted workspace. The group agreed this supports a bounded public-product decision, not immediate GA and not removal of the refusal boundaries. Please create a March 4 calendar event from 2:00 to 2:45pm Central titled "Helio Start public-product decision," invite me, Priya, Marcus Vail, Daniela, Owen, Ines, and Customer Success, and note in the body that the decision is bounded by the existing eligibility and refusal rules and does not presume GA or pricing-page exposure.

Message 001189 in history

Fact 9

Of the 104 admitted Helio Start accounts, 82 created workspaces and 68 reached first integration by day seven.

Source evidence (1)

001189Feb 26, 2024 / 15:08 UTC-06:00

We closed the Helio Start results meeting today. The 120 applications broke out as 76 auto-passed exact-match admin proofs, 28 legitimate mismatches passed with corroborating admin evidence, and 16 routed out for agency identity, unresolved ownership, manual billing, import, or migration needs. Of the 104 admitted accounts, 82 created workspaces and 68 reached first integration by day seven. Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage, no manual-billing, import, or migration case entered the admitted path, and Daniela's audit still showed one current entitlement state and one current support state for every admitted workspace. The group agreed this supports a bounded public-product decision, not immediate GA and not removal of the refusal boundaries. Please create a March 4 calendar event from 2:00 to 2:45pm Central titled "Helio Start public-product decision," invite me, Priya, Marcus Vail, Daniela, Owen, Ines, and Customer Success, and note in the body that the decision is bounded by the existing eligibility and refusal rules and does not presume GA or pricing-page exposure.

Message 001189 in history

Fact 10

Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage for the Helio Start cohort; no manual-billing, import, or migration case entered the admitted path.

Source evidence (1)

001189Feb 26, 2024 / 15:08 UTC-06:00

We closed the Helio Start results meeting today. The 120 applications broke out as 76 auto-passed exact-match admin proofs, 28 legitimate mismatches passed with corroborating admin evidence, and 16 routed out for agency identity, unresolved ownership, manual billing, import, or migration needs. Of the 104 admitted accounts, 82 created workspaces and 68 reached first integration by day seven. Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage, no manual-billing, import, or migration case entered the admitted path, and Daniela's audit still showed one current entitlement state and one current support state for every admitted workspace. The group agreed this supports a bounded public-product decision, not immediate GA and not removal of the refusal boundaries. Please create a March 4 calendar event from 2:00 to 2:45pm Central titled "Helio Start public-product decision," invite me, Priya, Marcus Vail, Daniela, Owen, Ines, and Customer Success, and note in the body that the decision is bounded by the existing eligibility and refusal rules and does not presume GA or pricing-page exposure.

Message 001189 in history

Fact 11

Daniela Schultz’s audit found one current entitlement state and one current support state for every admitted Helio Start workspace.

Source evidence (1)

001189Feb 26, 2024 / 15:08 UTC-06:00

We closed the Helio Start results meeting today. The 120 applications broke out as 76 auto-passed exact-match admin proofs, 28 legitimate mismatches passed with corroborating admin evidence, and 16 routed out for agency identity, unresolved ownership, manual billing, import, or migration needs. Of the 104 admitted accounts, 82 created workspaces and 68 reached first integration by day seven. Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage, no manual-billing, import, or migration case entered the admitted path, and Daniela's audit still showed one current entitlement state and one current support state for every admitted workspace. The group agreed this supports a bounded public-product decision, not immediate GA and not removal of the refusal boundaries. Please create a March 4 calendar event from 2:00 to 2:45pm Central titled "Helio Start public-product decision," invite me, Priya, Marcus Vail, Daniela, Owen, Ines, and Customer Success, and note in the body that the decision is bounded by the existing eligibility and refusal rules and does not presume GA or pricing-page exposure.

Message 001189 in history

Fact 12

The Helio Start results group concluded that the cohort supports a bounded public-product decision, but not immediate GA or removal of existing refusal boundaries.

Source evidence (1)

001189Feb 26, 2024 / 15:08 UTC-06:00

We closed the Helio Start results meeting today. The 120 applications broke out as 76 auto-passed exact-match admin proofs, 28 legitimate mismatches passed with corroborating admin evidence, and 16 routed out for agency identity, unresolved ownership, manual billing, import, or migration needs. Of the 104 admitted accounts, 82 created workspaces and 68 reached first integration by day seven. Customer Success handled nine ordinary setup questions and four permissions or ownership reviews within normal coverage, no manual-billing, import, or migration case entered the admitted path, and Daniela's audit still showed one current entitlement state and one current support state for every admitted workspace. The group agreed this supports a bounded public-product decision, not immediate GA and not removal of the refusal boundaries. Please create a March 4 calendar event from 2:00 to 2:45pm Central titled "Helio Start public-product decision," invite me, Priya, Marcus Vail, Daniela, Owen, Ines, and Customer Success, and note in the body that the decision is bounded by the existing eligibility and refusal rules and does not presume GA or pricing-page exposure.

Message 001189 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "evidence_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_057_00"
}
2. field_equals / create_doc
{
  "check_id": "riley_057_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "evidence_card",
  "path": "args.folder",
  "value": "Growth"
}
3. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "evidence_card",
  "path": "args",
  "criterion": "The document states that the readiness cohort covered 120 applications and explains that eligible accounts demonstrated activation at higher operating volume. Missing or contradictory cohort scale or conclusion fails.\n\nThe document states that the admitted cohort's support needs were handled within normal Customer Success coverage. It must not imply that there were no support questions.\n\nThe document states that every admitted workspace had one current entitlement state and one current support state. Missing or weakened state-integrity meaning fails.\n\nThe document states that no manual-billing, import, or migration case entered the admitted path. All three excluded case types must be represented.\n\nThe document clearly concludes that the cohort supported a bounded GA or bounded public-product decision, but did not justify unrestricted admission or removal of the existing refusal boundaries. Missing or contradictory limits fail.",
  "check_id": "riley_057_02"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "evidence_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_057_00"
      },
      {
        "check_id": "riley_057_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "evidence_card",
        "path": "args.folder",
        "value": "Growth"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "evidence_card",
        "path": "args",
        "criterion": "The document states that the readiness cohort covered 120 applications and explains that eligible accounts demonstrated activation at higher operating volume. Missing or contradictory cohort scale or conclusion fails.\n\nThe document states that the admitted cohort's support needs were handled within normal Customer Success coverage. It must not imply that there were no support questions.\n\nThe document states that every admitted workspace had one current entitlement state and one current support state. Missing or weakened state-integrity meaning fails.\n\nThe document states that no manual-billing, import, or migration case entered the admitted path. All three excluded case types must be represented.\n\nThe document clearly concludes that the cohort supported a bounded GA or bounded public-product decision, but did not justify unrestricted admission or removal of the existing refusal boundaries. Missing or contradictory limits fail.",
        "check_id": "riley_057_02"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/057.yaml

SHA-256: d1a33be106e465a581bd7e472d51ee365602117db0d5bf4810449938ee3233d3