Test 093

Jan 1, 2028 / 1 fact

Request

Create a focused, reusable review sheet for the new Expansion Assist lifecycle and in-product prompts. Center it on whether each message has a valid current-state reason to act rather than broad style guidance. Then DM Ines and ask her to apply the sheet to the draft set.

Required memory

Fact 33

Ines confirmed that the old day-60 nudge may be used only as copy material, and any Expansion Assist lifecycle or in-product message must describe the account’s current state rather than cite fixed signup age as the reason to act.

Source evidence (1)

001598Jun 12, 2024 / 11:35 UTC-05:00

Customer Success completed its individual review of the 46 behavior-flagged accounts and found nine with active adoption or renewal work that would make a commercial prompt inappropriate; those cases must be suppressed. Ines also confirmed that the old day-60 nudge is useful copy material only if any Expansion Assist lifecycle or in-product message describes the account's current state rather than using fixed signup age as the reason to act. Together with Owen's shadow counts, this completes the candidate-definition input for bounded pilot design, while the program remains observational and in shadow mode. I'm sharing the settled design input; no immediate action is needed.

Message 001598 in history

Expected tool calls

create_doc
send_slack_dm

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_review_sheet",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_093_00"
}

2. field_equals / send_slack_dm

{
  "type": "field_equals",
  "tool": "send_slack_dm",
  "action_id": "request_copy_review",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_093_01"
}

3. field_equals / send_slack_dm

{
  "type": "field_equals",
  "tool": "send_slack_dm",
  "action_id": "request_copy_review",
  "path": "args.user",
  "value": "ines",
  "check_id": "riley_093_02"
}

4. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_review_sheet",
  "path": "args",
  "criterion": "The body is a focused, reusable, actionable copy-review checklist or review sheet for evaluating Expansion Assist lifecycle and in-product messages, rather than merely prose commentary or broad general style advice.\n\nThe checklist explicitly requires reviewers to test each message against the account's current state and to reject or revise messaging whose reason to act relies merely on a fixed signup age or elapsed time since signup. Missing this distinction or endorsing time-since-signup as sufficient must fail.",
  "check_id": "riley_093_03"
}

5. field_llm_judge / send_slack_dm

{
  "type": "field_llm_judge",
  "tool": "send_slack_dm",
  "action_id": "request_copy_review",
  "path": "args",
  "criterion": "The message asks the recipient to apply the newly created copy-review sheet or checklist to the current draft set of Expansion Assist lifecycle and in-product prompts.",
  "check_id": "riley_093_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_review_sheet",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_093_00"
      },
      {
        "type": "field_equals",
        "tool": "send_slack_dm",
        "action_id": "request_copy_review",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_093_01"
      },
      {
        "type": "field_equals",
        "tool": "send_slack_dm",
        "action_id": "request_copy_review",
        "path": "args.user",
        "value": "ines",
        "check_id": "riley_093_02"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_review_sheet",
        "path": "args",
        "criterion": "The body is a focused, reusable, actionable copy-review checklist or review sheet for evaluating Expansion Assist lifecycle and in-product messages, rather than merely prose commentary or broad general style advice.\n\nThe checklist explicitly requires reviewers to test each message against the account's current state and to reject or revise messaging whose reason to act relies merely on a fixed signup age or elapsed time since signup. Missing this distinction or endorsing time-since-signup as sufficient must fail.",
        "check_id": "riley_093_03"
      },
      {
        "type": "field_llm_judge",
        "tool": "send_slack_dm",
        "action_id": "request_copy_review",
        "path": "args",
        "criterion": "The message asks the recipient to apply the newly created copy-review sheet or checklist to the current draft set of Expansion Assist lifecycle and in-product prompts.",
        "check_id": "riley_093_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/093.yaml

SHA-256: f7684e9cf121d8b6c184467bd894e90215678409bb72a1be9c62acd1422d58ae