Test 039

Jan 1, 2028 / 2 facts

Request

Create an archive note dated April 17, 2023, explaining the temporary second-post-ship cohort dip, the outage duration, and the system's condition after the incident. Do not include an exact outage start or end date.

Required memory

Fact 367

Daniela confirmed that an approximately 36-hour CS auto-onboarding-nudge outage caused the Apr 17 second-post-ship cohort dip; it was a one-time artifact, not an effect of the onboarding-trigger A/B.

Source evidence (1)

000187Apr 19, 2023 / 11:30 UTC-05:00

heads up from Daniela: the CS auto-onboarding-nudge system (her team owns it) was down ~36h on Apr 15-12. that's the root cause for the Apr 17 second-post-ship cohort dip to +1.4pts — one-time artifact, not the onboarding-trigger A/B. nudge is back up; she'll send the outage timeline. so the dip explanation is the CS auto-onboarding-nudge outage on Apr 15-12, full stop.

Message 000187 in history

Fact 368

The CS auto-onboarding-nudge system is back up, and Daniela will send the outage timeline.

Source evidence (1)

000187Apr 19, 2023 / 11:30 UTC-05:00

Message 000187 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_outage_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_039_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_outage_note",
  "path": "args",
  "criterion": "The note is dated April 17, 2023. It says an approximately 36-hour CS auto-onboarding-nudge outage caused the second-post-ship cohort dip; the dip was a one-time artifact rather than an effect of the onboarding-trigger A/B; and the system returned to service. It does not state an exact outage start or end date.",
  "check_id": "riley_039_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_outage_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_039_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_outage_note",
        "path": "args",
        "criterion": "The note is dated April 17, 2023. It says an approximately 36-hour CS auto-onboarding-nudge outage caused the second-post-ship cohort dip; the dip was a one-time artifact rather than an effect of the onboarding-trigger A/B; and the system returned to service. It does not state an exact outage start or end date.",
        "check_id": "riley_039_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/039.yaml

SHA-256: d6bfbe75c940e0d8a8d562db2b422c1a3612bf6dc9f31746473390120eae7043