DolphinBench
DiscordGitHub

Test 154

Jan 1, 2028 / 3 facts

YAML

Request

Create a standalone historical evidence card for the annual controls package covering the incomplete-setup reminder’s initial limited-production interval before expansion. Reconcile the total fresh evaluations across all four dispositions, including the reason for each non-display disposition, and record cross-surface agreement plus the customer-correction, incident, source-age, and rollback outcomes. Title it “Incomplete-setup reminder — initial production interval evidence card” and save it in the “Annual Controls” folder.

Required memory

Fact 285

On October 6, 2027 at 9:00 a.m. Central, Daniela released the incomplete-setup reminder to 10% of otherwise eligible production evaluations.

Source evidence (1)

004830Oct 6, 2027 / 10:08 UTC-05:00

Daniela released the incomplete-setup reminder at 9:00am to 10% of otherwise eligible production evaluations. Every display is freshly evaluated against current entitlement, qualifying-human-admin status, and active Customer Success work; active work suppresses, while missing, stale, unresolved, or conflicting state holds. The first 60 minutes completed without a cross-surface discrepancy, customer correction, source-age breach, failed rollback, or rollback use. The production-consumer count is now five rather than four.

Message 004830 in history

Fact 289

Owen closed the October 6–18 initial production interval for the incomplete-setup reminder.

Source evidence (1)

004896Oct 19, 2027 / 11:30 UTC-05:00

Owen closed the October 6–18 initial production interval for the incomplete-setup reminder at the unchanged 10% release. Across 3,240 fresh evaluations, the reminder displayed 412 times, 138 evaluations were suppressed because current Customer Success work was active, 61 were held for missing, stale, unresolved, or conflicting required state, and 2,629 produced no display because the current incomplete-setup or qualifying-human-admin condition was not met. Reporting, lifecycle, and reminder outcomes agreed for every evaluation. There was no customer correction, incident stop, source-age breach, rollback, or need for rollback. This completed interval supplies evidence for the separate rollout decision; it does not itself make that decision.

Message 004896 in history

Fact 290

During the incomplete-setup reminder's October 6–18 initial production interval, 3,240 fresh evaluations produced 412 displays, 138 suppressions due to active Customer Success work, 61 holds due to missing, stale, unresolved, or conflicting required state, and 2,629 non-displays because the incomplete-setup or qualifying-human-admin condition was not met; reporting, lifecycle, and reminder outcomes agreed for every evaluation, with no customer correction, incident stop, source-age breach, rollback, or need for rollback.

Source evidence (1)

004896Oct 19, 2027 / 11:30 UTC-05:00

Owen closed the October 6–18 initial production interval for the incomplete-setup reminder at the unchanged 10% release. Across 3,240 fresh evaluations, the reminder displayed 412 times, 138 evaluations were suppressed because current Customer Success work was active, 61 were held for missing, stale, unresolved, or conflicting required state, and 2,629 produced no display because the current incomplete-setup or qualifying-human-admin condition was not met. Reporting, lifecycle, and reminder outcomes agreed for every evaluation. There was no customer correction, incident stop, source-age breach, rollback, or need for rollback. This completed interval supplies evidence for the separate rollout decision; it does not itself make that decision.

Message 004896 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_154_00"
}
2. field_equals / create_doc
{
  "check_id": "riley_154_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "args.title",
  "value": "Incomplete-setup reminder — initial production interval evidence card"
}
3. field_equals / create_doc
{
  "check_id": "riley_154_02",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "args.folder",
  "value": "Annual Controls"
}
4. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_evidence_card",
  "path": "args",
  "criterion": "The body must identify the historical interval as October 6–18 at the unchanged 10% release and state that 3,240 fresh evaluations reconcile exactly across four distinct dispositions: 412 displayed; 138 suppressed because current Customer Success work was active; 61 held because required state was missing, stale, unresolved, or conflicting; and 2,629 produced no display because the current incomplete-setup or qualifying-human-admin condition was not met. The four counts must be presented as a complete reconciliation of the 3,240 evaluations, not conflated with one another.\n\nThe body must state that reporting, lifecycle, and reminder outcomes agreed for every evaluation and that the interval had no customer correction, no incident stop, no source-age breach, and no rollback. Wording that also says there was no need for rollback is acceptable.",
  "check_id": "riley_154_03"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_154_00"
      },
      {
        "check_id": "riley_154_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "args.title",
        "value": "Incomplete-setup reminder — initial production interval evidence card"
      },
      {
        "check_id": "riley_154_02",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "args.folder",
        "value": "Annual Controls"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_evidence_card",
        "path": "args",
        "criterion": "The body must identify the historical interval as October 6–18 at the unchanged 10% release and state that 3,240 fresh evaluations reconcile exactly across four distinct dispositions: 412 displayed; 138 suppressed because current Customer Success work was active; 61 held because required state was missing, stale, unresolved, or conflicting; and 2,629 produced no display because the current incomplete-setup or qualifying-human-admin condition was not met. The four counts must be presented as a complete reconciliation of the 3,240 evaluations, not conflated with one another.\n\nThe body must state that reporting, lifecycle, and reminder outcomes agreed for every evaluation and that the interval had no customer correction, no incident stop, no source-age breach, and no rollback. Wording that also says there was no need for rollback is acceptable.",
        "check_id": "riley_154_03"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/154.yaml

SHA-256: b64ac577770cdfb10a4332665530bffc45fb0d2fbdc4aabb6302e12f24e89bcd