Test 073

Jan 1, 2028 / 2 facts

Request

Create a facilitator sheet for Hema's quarterly Infra onboarding module using its three adopted incident-practice scenarios and scoring rules.

Required memory

Fact 147

Alex Valdez requested a comment on the incident-practice document recording that, in the January 24 exercise, five of eight participants paged the secondary before the 45-minute boundary when the cause remained unisolated, three of eight treated a Friday staging rehearsal with no production-state change as a prohibited deploy, and four of eight initially stopped the postmortem at the triggering operator command instead of identifying the absent validation guardrail.

Source evidence (1)

002341Jan 24, 2025 / 11:20 UTC-05:00

Post a comment on the incident-practice document recording these January exercise results: five of eight participants paged the secondary before the 45-minute boundary when the cause remained unisolated; three of eight still treated a Friday staging rehearsal with no production-state change as a prohibited deploy; and four of eight initially stopped the postmortem at the triggering operator command instead of naming the absent validation guardrail. Record the revised February 14 test requiring every response to state the incident clock and secondary timing, whether the change affects production state, and the absent system control or validation guardrail.

Message 002341 in history

Fact 148

Hema adopted the incident-practice scenarios as a quarterly infrastructure-rotation onboarding module; Wes will teach the responder path, Nadia will review the postmortem section, and Alex Valdez will maintain the cross-service failure cases.

Source evidence (1)

002401Feb 14, 2025 / 11:18 UTC-05:00

The revised timed exercise finished with all eight participants engaging the secondary before the 45-minute boundary when isolation failed. All eight correctly distinguished Friday staging preparation with no production-state change from a production change, and seven of eight identified the missing system guardrail without framing the triggering operator action as root cause. This uses the scoring correction I posted on February 4. Hema adopted the scenarios as a quarterly infra-rotation onboarding module: Wes will teach the responder path, Nadia will review the postmortem section, and I will maintain the cross-service failure cases. The assigned February exercise is complete.

Message 002401 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "alex_073_create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_073_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "The facilitator sheet includes timed scenarios and scoring for three behaviors: engage the secondary before 45 minutes when isolation fails; distinguish Friday staging preparation with no production-state change from a production deploy; and identify the missing system control or validation guardrail rather than naming only the triggering operator action as root cause.",
  "tool": "create_doc",
  "action_id": "alex_073_create_doc",
  "check_id": "alex_073_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "alex_073_create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_073_00"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "The facilitator sheet includes timed scenarios and scoring for three behaviors: engage the secondary before 45 minutes when isolation fails; distinguish Friday staging preparation with no production-state change from a production deploy; and identify the missing system control or validation guardrail rather than naming only the triggering operator action as root cause.",
        "tool": "create_doc",
        "action_id": "alex_073_create_doc",
        "check_id": "alex_073_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/073.yaml

SHA-256: d9727da5fa912fbf6162ab0c3474b0eb98bf9afe2435788937f6ec74a1ff1132