Test 170

Jan 1, 2028 / 5 facts

Request

Create a concise, denominator-first evidence card in the “Connector-access-check archive” for future Helio Start operating reviews. Summarize what the first fully mature post-check cohort established about credential-related failures, and clearly separate that operational finding from any longer-horizon business outcomes the cohort did not establish.

Required memory

Fact 28

Owen closed the first fully mature cohort after the connector-access check.

Source evidence (1)

001515May 20, 2024 / 09:10 UTC-05:00

Owen closed the first fully mature cohort after the connector-access check. Of 92 completed applications, 68 were admitted, 15 stayed pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome. Fifty-nine of the 68 admitted workspaces completed a supported first integration within seven days. Three of the nine nonactivators were traced to connector credentials, compared with 18 credential-related stalls among the prior 147 admitted workspaces. Customer Success recorded 11 support contacts across eight admitted workspaces. Calculate the admission and activation rates, and compare the credential-related stalls using the proper admitted denominators. Then give me a denominator-first interpretation that supports the intake check without claiming mature retention, paid conversion, or revenue performance.

Message 001515 in history

Fact 29

In the first fully mature cohort after the connector-access check, 68 of 92 completed applications were admitted; 15 remained pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome.

Source evidence (1)

001515May 20, 2024 / 09:10 UTC-05:00

Owen closed the first fully mature cohort after the connector-access check. Of 92 completed applications, 68 were admitted, 15 stayed pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome. Fifty-nine of the 68 admitted workspaces completed a supported first integration within seven days. Three of the nine nonactivators were traced to connector credentials, compared with 18 credential-related stalls among the prior 147 admitted workspaces. Customer Success recorded 11 support contacts across eight admitted workspaces. Calculate the admission and activation rates, and compare the credential-related stalls using the proper admitted denominators. Then give me a denominator-first interpretation that supports the intake check without claiming mature retention, paid conversion, or revenue performance.

Message 001515 in history

Fact 30

In the first fully mature cohort after the connector-access check, 59 of 68 admitted workspaces completed a supported first integration within seven days, and three of the nine nonactivators were traced to connector credentials.

Source evidence (1)

001515May 20, 2024 / 09:10 UTC-05:00

Owen closed the first fully mature cohort after the connector-access check. Of 92 completed applications, 68 were admitted, 15 stayed pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome. Fifty-nine of the 68 admitted workspaces completed a supported first integration within seven days. Three of the nine nonactivators were traced to connector credentials, compared with 18 credential-related stalls among the prior 147 admitted workspaces. Customer Success recorded 11 support contacts across eight admitted workspaces. Calculate the admission and activation rates, and compare the credential-related stalls using the proper admitted denominators. Then give me a denominator-first interpretation that supports the intake check without claiming mature retention, paid conversion, or revenue performance.

Message 001515 in history

Fact 31

Connector credentials caused 18 stalls among the prior 147 admitted workspaces.

Source evidence (1)

001515May 20, 2024 / 09:10 UTC-05:00

Owen closed the first fully mature cohort after the connector-access check. Of 92 completed applications, 68 were admitted, 15 stayed pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome. Fifty-nine of the 68 admitted workspaces completed a supported first integration within seven days. Three of the nine nonactivators were traced to connector credentials, compared with 18 credential-related stalls among the prior 147 admitted workspaces. Customer Success recorded 11 support contacts across eight admitted workspaces. Calculate the admission and activation rates, and compare the credential-related stalls using the proper admitted denominators. Then give me a denominator-first interpretation that supports the intake check without claiming mature retention, paid conversion, or revenue performance.

Message 001515 in history

Fact 32

Riley Tanaka requested a denominator-first interpretation that supports the connector-access intake check without claiming mature retention, paid conversion, or revenue performance.

Source evidence (1)

001515May 20, 2024 / 09:10 UTC-05:00

Owen closed the first fully mature cohort after the connector-access check. Of 92 completed applications, 68 were admitted, 15 stayed pending or entered assistance because connector authority was missing or uncertain, and nine had another assisted or ineligible outcome. Fifty-nine of the 68 admitted workspaces completed a supported first integration within seven days. Three of the nine nonactivators were traced to connector credentials, compared with 18 credential-related stalls among the prior 147 admitted workspaces. Customer Success recorded 11 support contacts across eight admitted workspaces. Calculate the admission and activation rates, and compare the credential-related stalls using the proper admitted denominators. Then give me a denominator-first interpretation that supports the intake check without claiming mature retention, paid conversion, or revenue performance.

Message 001515 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_connector_evidence_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_170_00"
}

2. field_equals / create_doc

{
  "check_id": "riley_170_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_connector_evidence_card",
  "path": "args.folder",
  "value": "Connector-access-check archive"
}

3. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_connector_evidence_card",
  "path": "args",
  "criterion": "The body gives a denominator-first comparison for credential-related failures among admitted workspaces: 3 of 68 in the first fully mature cohort after the connector-access check (about 4.4%) versus 18 of 147 in the prior cohort (about 12.2%). It accurately interprets the lower post-check rate as operational evidence supporting the connector-access check, without presenting the comparison as proof of broader causality.\n\nThe body clearly and explicitly states that the mature-cohort result did not establish mature retention, paid conversion, or revenue performance. Missing any of these three limitations, or claiming that any was established, must fail.",
  "check_id": "riley_170_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_connector_evidence_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_170_00"
      },
      {
        "check_id": "riley_170_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_connector_evidence_card",
        "path": "args.folder",
        "value": "Connector-access-check archive"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_connector_evidence_card",
        "path": "args",
        "criterion": "The body gives a denominator-first comparison for credential-related failures among admitted workspaces: 3 of 68 in the first fully mature cohort after the connector-access check (about 4.4%) versus 18 of 147 in the prior cohort (about 12.2%). It accurately interprets the lower post-check rate as operational evidence supporting the connector-access check, without presenting the comparison as proof of broader causality.\n\nThe body clearly and explicitly states that the mature-cohort result did not establish mature retention, paid conversion, or revenue performance. Missing any of these three limitations, or claiming that any was established, must fail.",
        "check_id": "riley_170_02"
      }
    ]
  }
}

App state

Source file

tests/riley/170.yaml

SHA-256: ca9d719be8c66b705c203aeb1aa9152fdedced17a20c0e0c25088d1d76c6e5eb