Test 107

Jan 1, 2028 / 1 fact

Request

Please review the legacy-trigger description in document `doc_lifecycle_trigger_migration_2028` and post a correction comment that states how the original “See how teams use Helio” email actually worked. Clearly label the corrected behavior as the pre-v2 configuration. Do not edit the document itself.

Required memory

Fact 325

In v1, the onboarding email "See how teams use Helio" is sent exactly seven calendar days after signup without checking activation state, so activated and unactivated users receive it on the same schedule.

Source evidence (1)

000053Feb 22, 2023 / 14:00 UTC-06:00

Ines walked me through the lifecycle setup. v1 is a little janky tbh. The day-7 email -- "See how teams use Helio" -- is just calendar-day off signup, exact day 7, no check for activation state at all. So activated and totally cold users both get the same thing on the same clock. That's probably fine if all we're testing is top-of-funnel education, but if we're trying to explain the mid-seg churn wobble I don't love that we have a state-blind touch sitting right there in onboarding. Need the send volume + open/click split by activated vs not before I decide if it's real or just ugly config.

Message 000053 in history

Expected tool calls

post_doc_comment

Grading

1. field_equals / post_doc_comment

{
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_legacy_trigger_correction",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_107_00"
}

2. field_equals / post_doc_comment

{
  "check_id": "riley_107_01",
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_legacy_trigger_correction",
  "path": "args.doc_id",
  "value": "doc_lifecycle_trigger_migration_2028"
}

3. field_llm_judge / post_doc_comment

{
  "type": "field_llm_judge",
  "tool": "post_doc_comment",
  "action_id": "post_legacy_trigger_correction",
  "path": "args",
  "criterion": "The comment must clearly correct the draft by explaining that the legacy v1 / pre-v2 configuration for “See how teams use Helio” sent exactly on calendar day 7 after signup, performed no activation-state check, and therefore sent to both activated and not-yet-activated users on the same day-7 clock. Missing, ambiguous, or contradictory descriptions must fail.",
  "check_id": "riley_107_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_legacy_trigger_correction",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_107_00"
      },
      {
        "check_id": "riley_107_01",
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_legacy_trigger_correction",
        "path": "args.doc_id",
        "value": "doc_lifecycle_trigger_migration_2028"
      },
      {
        "type": "field_llm_judge",
        "tool": "post_doc_comment",
        "action_id": "post_legacy_trigger_correction",
        "path": "args",
        "criterion": "The comment must clearly correct the draft by explaining that the legacy v1 / pre-v2 configuration for “See how teams use Helio” sent exactly on calendar day 7 after signup, performed no activation-state check, and therefore sent to both activated and not-yet-activated users on the same day-7 clock. Missing, ambiguous, or contradictory descriptions must fail.",
        "check_id": "riley_107_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/107.yaml

SHA-256: b4c1c673875be9fe569dd1815dc91863c2f65597be88d69af3c6b8d8f2be77dd