Test 037

Jan 1, 2028 / 1 fact

Request

Create a one-metric archive card for the April 25 third post-ship mid-segment cohort, then post a link to the card in #growth-investigations. Include only the April 25 numerical result, not an earlier numerical read.

Required memory

Fact 373

The Apr. 25 third post-ship mid-segment cohort cut showed week-1 retention 3.4 percentage points above baseline, exceeding the first post-ship cohort’s 3.2-point lift.

Source evidence (1)

000202Apr 25, 2023 / 11:00 UTC-05:00

third post-ship cut came in clean. Apr 25 week-1 retention on the mid-seg cohort is +3.4pts vs baseline, so better than the first +3.2 read and not doing that annoying roll-over thing after the Apr 17 wobble. I dropped it in #growth-investigations and, small thing but not small, called it "the trend" for the first time since Feb instead of "the signal." feels like the 75-200 recut plus backing out the CS auto-nudge mess got us to the real read. still gonna watch another week obviously, but this is the first one that feels actually solid.

Message 000202 in history

Expected tool calls

create_doc
send_slack_message

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_archive_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_037_00"
}

2. field_equals / send_slack_message

{
  "type": "field_equals",
  "tool": "send_slack_message",
  "action_id": "post_archive_pointer",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_037_01"
}

3. field_equals / send_slack_message

{
  "check_id": "riley_037_02",
  "type": "field_equals",
  "tool": "send_slack_message",
  "action_id": "post_archive_pointer",
  "path": "args.channel",
  "value": "#growth-investigations"
}

4. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_archive_card",
  "path": "args",
  "criterion": "The document is a compact archival card that clearly identifies the Apr 25 third post-ship mid-segment cohort and reports week-1 retention as +3.4 points versus baseline. It must treat that as the card's sole measured performance result; missing or contradictory cohort, metric, value, or comparison information fails.",
  "check_id": "riley_037_03"
}

5. field_llm_judge / send_slack_message

{
  "type": "field_llm_judge",
  "tool": "send_slack_message",
  "action_id": "post_archive_pointer",
  "path": "args",
  "criterion": "The message is a concise, discoverable pointer to the newly created archive card for the third post-ship cohort and contains a usable link to that document. A message without a document link or without enough context to identify the card fails.",
  "check_id": "riley_037_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_archive_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_037_00"
      },
      {
        "type": "field_equals",
        "tool": "send_slack_message",
        "action_id": "post_archive_pointer",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_037_01"
      },
      {
        "check_id": "riley_037_02",
        "type": "field_equals",
        "tool": "send_slack_message",
        "action_id": "post_archive_pointer",
        "path": "args.channel",
        "value": "#growth-investigations"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_archive_card",
        "path": "args",
        "criterion": "The document is a compact archival card that clearly identifies the Apr 25 third post-ship mid-segment cohort and reports week-1 retention as +3.4 points versus baseline. It must treat that as the card's sole measured performance result; missing or contradictory cohort, metric, value, or comparison information fails.",
        "check_id": "riley_037_03"
      },
      {
        "type": "field_llm_judge",
        "tool": "send_slack_message",
        "action_id": "post_archive_pointer",
        "path": "args",
        "criterion": "The message is a concise, discoverable pointer to the newly created archive card for the third post-ship cohort and contains a usable link to that document. A message without a document link or without enough context to identify the card fails.",
        "check_id": "riley_037_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/037.yaml

SHA-256: 87db69d6b04cac57e2ae200d0f2fb8b7e979784ab1bad83d2d09890da092ff0c