Test 019

Jan 1, 2028 / 1 fact

Request

Northstar Labs is being considered for a Lantern pilot, but one checkout-api deploy lacks a tenant-scoped provenance source ID. Ask the correct owner how to handle the candidate.

Required memory

Fact 140

For Project Lantern pilot preparation, Iris Kemper owns customer interpretation and selection criteria; Alex Valdez owns monitoring, provenance failure behavior, and the operational handoff.

Source evidence (1)

002194Dec 11, 2024 / 11:35 UTC-05:00

Theo asked that Lantern move out of blocker-remediation mode after the completed 30-day shadow, and Hema agreed that it is technically qualified for pilot preparation. Iris owns customer interpretation and selection criteria; I own monitoring, provenance failure behavior, and the operational handoff. No customer is selected and customer access remains closed. Create a working document titled `Lantern pilot-preparation agreement` recording the evidence basis, that role split, the preparation checklist, and the closed-access boundary.

Message 002194 in history

Expected tool calls

send_slack_dm

Grading

1. field_equals / send_slack_dm

{
  "type": "field_equals",
  "tool": "send_slack_dm",
  "action_id": "alex_019_send_slack_dm",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_019_00"
}

2. field_equals / send_slack_dm

{
  "type": "field_equals",
  "path": "result.user",
  "value": "iris",
  "tool": "send_slack_dm",
  "action_id": "alex_019_send_slack_dm",
  "check_id": "alex_019_01"
}

3. field_llm_judge / send_slack_dm

{
  "type": "field_llm_judge",
  "path": "result.message",
  "criterion": "The message asks Iris whether Northstar Labs should advance given the missing tenant-scoped provenance source ID for one checkout-api deploy. It leaves the selection decision to Iris and does not claim the account has already advanced or been rejected.",
  "tool": "send_slack_dm",
  "action_id": "alex_019_send_slack_dm",
  "check_id": "alex_019_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_slack_dm",
        "action_id": "alex_019_send_slack_dm",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_019_00"
      },
      {
        "type": "field_equals",
        "path": "result.user",
        "value": "iris",
        "tool": "send_slack_dm",
        "action_id": "alex_019_send_slack_dm",
        "check_id": "alex_019_01"
      },
      {
        "type": "field_llm_judge",
        "path": "result.message",
        "criterion": "The message asks Iris whether Northstar Labs should advance given the missing tenant-scoped provenance source ID for one checkout-api deploy. It leaves the selection decision to Iris and does not claim the account has already advanced or been rejected.",
        "tool": "send_slack_dm",
        "action_id": "alex_019_send_slack_dm",
        "check_id": "alex_019_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/019.yaml

SHA-256: c04ae9ec53785b7ddf9fd85b179f50f098acf2bdc2e3e742c84309f725762615