Test 058 for Morgan Chen

Required memory

Fact 104

Under Mercury’s May 24, 2023 activation mapping, an account is activated within seven days only if real_source_connected or first_live_sync_completed occurs; sample_import_completed does not count, and invite_sent is supporting onboarding evidence but does not establish activation by itself.

Source evidence (1)

000534May 24, 2023 / 13:38 UTC-07:00

Anna dropped the canonical correction from the review. Discord group DM — Anna Martinez, Morgan Chen, Devon Hayes 2023-05-24, 1:24 PM Anna Martinez: Dropping the corrected note from the review here so we have one canonical version. --- Looker note Title: Mercury activation mapping correction — 2023-05-24 Owner: Anna Martinez Audience: internal only Summary The prior weekly activation cut was inflated by demo-importer / sample-import activity. Accounts that completed sample data flows were being counted alongside accounts that actually connected a real source or completed a first live sync. That is too loose for activation. Directional cohort check | Cohort window | Accounts created | Prior activation cut | Corrected activation cut | Inflation source | | --- | ---: | ---: | ---: | --- | | 2023-05-08 to 2023-05-14 | 96 | 41 (42.7%) | 31 (32.3%) | 10 accounts were sample/demo-path only | | 2023-05-15 to 2023-05-21 | 88 | 36 (40.9%) | 27 (30.7%) | 9 accounts were sample/demo-path only | What changed - demo_importer and related sample flows were pulling sample_import_completed into the activation logic - that caused accounts to look activated before any real source was connected - invite activity correlates with onboarding progress but does not by itself establish activation Final mapping for Mercury activation - Count as activated within 7 days if either of these occurs: - real_source_connected - first_live_sync_completed - Do not count: - sample_import_completed - Supporting evidence only, not activation by itself: - invite_sent Notes - This corrected cut is appropriate for internal operating use - Do not use this weekly cut on board or external charts - For board / investor retention and activation framing, stay on the sturdier cohort view rather than this weekly directional slice --- Anna Martinez: Net: the shape is still useful internally, but the earlier weekly number was definitely too generous because of sample/demo activity. Save a new internal note titled `Mercury activation reporting definition — May 24 correction`, then post a short direction in the active team chat. Use Anna’s version as the source of truth, including the definition and the audience limits.

Message 000534 in history

Grading

1. field_equals / post_pr_comment

{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_058_00",
  "action_id": "morgan_058_post_pr_comment"
}

2. field_equals / post_pr_comment

{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/analytics#6214",
  "tool": "post_pr_comment",
  "check_id": "morgan_058_01",
  "action_id": "morgan_058_post_pr_comment"
}

3. field_llm_judge / post_pr_comment

{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Counts real_source_connected or first_live_sync_completed within 7 days as activation. Excludes sample_import_completed and does not let invite_sent alone activate an account. Describing invitations as supporting activity is valid but the literal phrase supporting evidence is unnecessary.",
  "tool": "post_pr_comment",
  "check_id": "morgan_058_02",
  "action_id": "morgan_058_post_pr_comment"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_058_00",
        "action_id": "morgan_058_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/analytics#6214",
        "tool": "post_pr_comment",
        "check_id": "morgan_058_01",
        "action_id": "morgan_058_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Counts real_source_connected or first_live_sync_completed within 7 days as activation. Excludes sample_import_completed and does not let invite_sent alone activate an account. Describing invitations as supporting activity is valid but the literal phrase supporting evidence is unnecessary.",
        "tool": "post_pr_comment",
        "check_id": "morgan_058_02",
        "action_id": "morgan_058_post_pr_comment"
      }
    ]
  }
}

Test 058

Request

Required memory

Fact 104

Source evidence (1)

Expected tool calls

Grading