DolphinBench

Test 160

Jan 1, 2028 / 5 facts

YAML

Request

Create an audit-ready evidence card for the historical CS auto-nudge production shadow. Include the interval, reconciled outcomes and causes, suppressions, reviewer validations, release boundary, and a plain explanation of external-system authority. Distinguish shadow evaluation from a production switch.

Required memory

Fact 102

The CS auto-nudge team scheduled a production shadow for April 25 through May 2, 2025; the legacy path remains the released production path during the shadow.

Source evidence (1)

002871Apr 24, 2025 / 15:18 UTC-05:00

Daniela completed the historical replay of 412 recent CS auto-nudge decisions against the shared state contract. 386 matched the legacy result. The contract prevented the remaining 26 from proceeding under the legacy eligibility result: 19 became field-specific holds - 15 for stale qualifying-human-admin state and four for unresolved external-system authority - and seven became suppressions because current Customer Success adoption or renewal work was active. No record with all required state current and supported changed outcome. Owen opened with the denominator and operating consequence, and I joined only for the release boundary. The team scheduled a production shadow from April 25 through May 2, while the legacy path remains the released production path. Finalize the existing replay brief with the total and the operational category breakdown, and add the production-shadow work section without implying a production switch.

Message 002871 in history

Fact 103

The April 25-May 2, 2025 CS auto-nudge production shadow completed 204 live evaluations on both paths without changing the released audience: 191 decisions agreed, while the shared contract prevented 13 from proceeding under the legacy eligibility result. Nine became field-specific holds—seven for stale qualifying-human-admin state and two for unresolved external-system authority—and four became suppressions because current Customer Success adoption or renewal work was active; every difference followed an established contract rule, with no unmatched or unclassified cases.

Source evidence (1)

002897May 2, 2025 / 14:43 UTC-05:00

The April 25-May 2 CS auto-nudge production shadow is complete. Both paths evaluated 204 live decisions without changing the released audience: 191 agreed, and the shared contract prevented 13 from proceeding under the legacy eligibility result. Nine became field-specific holds - seven for stale qualifying-human-admin state and two for unresolved external-system authority - while four became suppressions because current Customer Success adoption or renewal work was active. Daniela verified the hold and suppression reasons and production rollback behavior. Ines confirmed that no held or suppressed case received adjacent lifecycle messaging that assumed eligibility. Every difference follows an established contract rule, with no unmatched or unclassified case. I authorize a bounded May 8 production release, but the legacy path remains the released production path until that switch occurs. Finalize the production-shadow brief with the full evidence and send the bounded authorization to Lifecycle Systems and Customer Success without describing the integration as live yet.

Message 002897 in history

Fact 104

Daniela verified the CS auto-nudge production shadow's hold and suppression reasons and production rollback behavior.

Source evidence (1)

002897May 2, 2025 / 14:43 UTC-05:00

The April 25-May 2 CS auto-nudge production shadow is complete. Both paths evaluated 204 live decisions without changing the released audience: 191 agreed, and the shared contract prevented 13 from proceeding under the legacy eligibility result. Nine became field-specific holds - seven for stale qualifying-human-admin state and two for unresolved external-system authority - while four became suppressions because current Customer Success adoption or renewal work was active. Daniela verified the hold and suppression reasons and production rollback behavior. Ines confirmed that no held or suppressed case received adjacent lifecycle messaging that assumed eligibility. Every difference follows an established contract rule, with no unmatched or unclassified case. I authorize a bounded May 8 production release, but the legacy path remains the released production path until that switch occurs. Finalize the production-shadow brief with the full evidence and send the bounded authorization to Lifecycle Systems and Customer Success without describing the integration as live yet.

Message 002897 in history

Fact 105

Ines confirmed that no case held or suppressed during the CS auto-nudge production shadow received adjacent lifecycle messaging that assumed eligibility.

Source evidence (1)

002897May 2, 2025 / 14:43 UTC-05:00

The April 25-May 2 CS auto-nudge production shadow is complete. Both paths evaluated 204 live decisions without changing the released audience: 191 agreed, and the shared contract prevented 13 from proceeding under the legacy eligibility result. Nine became field-specific holds - seven for stale qualifying-human-admin state and two for unresolved external-system authority - while four became suppressions because current Customer Success adoption or renewal work was active. Daniela verified the hold and suppression reasons and production rollback behavior. Ines confirmed that no held or suppressed case received adjacent lifecycle messaging that assumed eligibility. Every difference follows an established contract rule, with no unmatched or unclassified case. I authorize a bounded May 8 production release, but the legacy path remains the released production path until that switch occurs. Finalize the production-shadow brief with the full evidence and send the bounded authorization to Lifecycle Systems and Customer Success without describing the integration as live yet.

Message 002897 in history

Fact 114

No evaluated decision had an external-system-authority, connector-authentication, entitlement, qualifying-human-admin, or Customer Success-work status that differed from the corresponding reporting and lifecycle views; for this review, external-system authority means verified readiness to perform the connector's required owner-level permission or network-allowlist action, not permission for automated contact.

Source evidence (1)

003012Jun 13, 2025 / 14:30 UTC-05:00

The formal first-live CS auto-nudge operating review is complete. Close the living review note with the accepted result and authority boundaries, send the participant summary, and create the June 17 implementation checkpoint from 10:00am to 10:30am Central with me, Owen, Ines, Daniela, and Customer Success. Owen closed the ledger at 1,184 unique evaluated decisions. Fifty-three stale or conflicting decisions failed closed, including the 12 affected by the May 20 source delay, and 1,131 proceeded or were suppressed using current state. Holds and fresh-check outcomes remain in audit history without adding to the primary denominator. No decision had a different external-system-authority, connector-authentication, entitlement, qualifying-human-admin, or Customer Success-work status from the corresponding reporting and lifecycle views, and no customer correction was required. External-system authority remains verified readiness to perform the connector’s required owner-level permission or network-allowlist action, not permission for automated contact. Priya accepted Owen’s recommendation to retire the parallel legacy decision path on June 17. The path remains available through that scheduled change and has not been retired early; Daniela’s engineering rollback control remains intact. Owen owns routine operating explanation. I retain disputed semantics, cross-surface consequences, customer-correction and incident-stop decisions, and executive claims. The ledger, owner sections, and my final pass are complete, and Priya’s decision is made.

Message 003012 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_shadow_evidence_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_160_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_shadow_evidence_card",
  "path": "args",
  "criterion": "The title clearly identifies the document as an evidence card or audit record for the CS auto-nudge production shadow. A generic controls document or blank-template title fails.\n\nThe body states that the production shadow ran from April 25 through May 2, 2025, that both paths evaluated live decisions, and that the shadow did not change the released audience. It must present this as a pre-release shadow gate rather than claiming the integration switched production during the shadow; missing or contradictory boundaries fail.\n\nThe body reconciles the complete shadow population: 204 live decisions total, 191 agreements, and 13 decisions prevented from proceeding under the legacy eligibility result, with those 13 further reconciling to nine field-specific holds plus four suppressions. Missing, inconsistent, or contradictory totals fail.\n\nThe body breaks down the nine field-specific holds into exactly seven caused by stale qualifying-human-admin state and two caused by unresolved external-system authority for the connector's required owner-level permission or network-allowlist action. Missing causes, wrong counts, or a materially different meaning of external-system authority fail.\n\nThe body states that exactly four decisions were suppressed because current Customer Success adoption or renewal work was active. Missing, contradictory, or differently attributed suppressions fail.\n\nThe body records that Daniela verified the field-specific hold and suppression reasons and production rollback behavior, and that Ines confirmed no held or suppressed case received adjacent lifecycle messaging that assumed eligibility. Both reviewer findings must be present and correctly attributed; missing or contradicted findings fail.\n\nThe body states that every observed difference followed an established shared-contract rule and that there were no unmatched or unclassified cases. Missing or contradictory coverage findings fail.",
  "check_id": "riley_160_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_shadow_evidence_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_160_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_shadow_evidence_card",
        "path": "args",
        "criterion": "The title clearly identifies the document as an evidence card or audit record for the CS auto-nudge production shadow. A generic controls document or blank-template title fails.\n\nThe body states that the production shadow ran from April 25 through May 2, 2025, that both paths evaluated live decisions, and that the shadow did not change the released audience. It must present this as a pre-release shadow gate rather than claiming the integration switched production during the shadow; missing or contradictory boundaries fail.\n\nThe body reconciles the complete shadow population: 204 live decisions total, 191 agreements, and 13 decisions prevented from proceeding under the legacy eligibility result, with those 13 further reconciling to nine field-specific holds plus four suppressions. Missing, inconsistent, or contradictory totals fail.\n\nThe body breaks down the nine field-specific holds into exactly seven caused by stale qualifying-human-admin state and two caused by unresolved external-system authority for the connector's required owner-level permission or network-allowlist action. Missing causes, wrong counts, or a materially different meaning of external-system authority fail.\n\nThe body states that exactly four decisions were suppressed because current Customer Success adoption or renewal work was active. Missing, contradictory, or differently attributed suppressions fail.\n\nThe body records that Daniela verified the field-specific hold and suppression reasons and production rollback behavior, and that Ines confirmed no held or suppressed case received adjacent lifecycle messaging that assumed eligibility. Both reviewer findings must be present and correctly attributed; missing or contradicted findings fail.\n\nThe body states that every observed difference followed an established shared-contract rule and that there were no unmatched or unclassified cases. Missing or contradictory coverage findings fail.",
        "check_id": "riley_160_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/160.yaml

SHA-256: 9c88613a48378f6c318f690e3cee95ebfb7d7c7be2e67e4c2cbec7808e25686f