Test 159

Jan 1, 2028 / 2 facts

Request

Create a compact 2028 lifecycle QA specification document for the entitlement-name regression coverage in the renewal reminder, trial-expiry education, and integration summary templates. Cover populated and missing entitlement-name cases, the authoritative rendering source, fallback behavior, and the legacy source that must never be referenced.

Required memory

Fact 90

Daniela deployed the engineering-controlled Customer.io personalization mapping for the renewal reminder, trial-expiry education, and integration summary.

Source evidence (1)

002795Apr 3, 2025 / 09:25 UTC-05:00

Daniela deployed the engineering-controlled Customer.io personalization mapping for the renewal reminder, trial-expiry education, and integration summary. Ines verified every released production path with populated and missing entitlement fixtures: populated records render `entitlement.display_name`, missing values render `your plan`, and none of the three active templates still references `workspace.plan_name`. The governed entitlement field and explicit fallback are now live in all three templates, so the migration is complete before the April 15 deprecation.

Message 002795 in history

Fact 91

Ines verified every released production path for the three Customer.io templates using populated and missing entitlement fixtures: populated records render `entitlement.display_name`, missing values render `your plan`, and no active template references `workspace.plan_name`.

Source evidence (1)

002795Apr 3, 2025 / 09:25 UTC-05:00

Message 002795 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_2028_lifecycle_qa_spec",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_159_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_2028_lifecycle_qa_spec",
  "path": "args",
  "criterion": "The title clearly identifies the document as a 2028 lifecycle entitlement-name regression or QA specification.\n\nThe body provides QA coverage for each of these three templates: renewal reminder, trial-expiry education, and integration summary. For every template, it specifies that a populated entitlement name renders from the governed or authoritative field `entitlement.display_name`, a missing value renders the exact fallback `your plan`, and the template must never reference the legacy field `workspace.plan_name`. It also makes clear that entitlement-name changes must flow through the governed entitlement field. Missing or contradictory mappings fail this check.",
  "check_id": "riley_159_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_2028_lifecycle_qa_spec",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_159_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_2028_lifecycle_qa_spec",
        "path": "args",
        "criterion": "The title clearly identifies the document as a 2028 lifecycle entitlement-name regression or QA specification.\n\nThe body provides QA coverage for each of these three templates: renewal reminder, trial-expiry education, and integration summary. For every template, it specifies that a populated entitlement name renders from the governed or authoritative field `entitlement.display_name`, a missing value renders the exact fallback `your plan`, and the template must never reference the legacy field `workspace.plan_name`. It also makes clear that entitlement-name changes must flow through the governed entitlement field. Missing or contradictory mappings fail this check.",
        "check_id": "riley_159_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/159.yaml

SHA-256: 032f295c585e572bcfd22e0addb5f399c616809a54e7db42d0701f478f8a9f91