Test 053

Jan 1, 2028 / 1 fact

Request

Inspect the shard-keeper label-contract runbook and update it to the production key deployed today.

Required memory

Fact 108

Roman completed the production rollout of the shard-keeper label rename: shard-keeper now emits `storage_region`, rollup-service reads it, and one hour of production observation showed matching label counts, zero dropped rollups, normal rollup completion, and no change to alert semantics.

Source evidence (1)

001139Feb 20, 2024 / 15:50 UTC-05:00

Roman finished the production rollout. Shard-keeper now emits `storage_region`, rollup-service reads that key, and an hour of production observation shows matching label counts, zero dropped rollups, normal rollup completion, and no alert-semantic change. The shared fixture still contains the executable old-to-new parity case that caught the original mismatch, so the blocked rename has now landed safely.

Message 001139 in history

Expected tool calls

get_runbook
update_runbook_entry

Grading

1. field_equals / update_runbook_entry

{
  "type": "field_equals",
  "tool": "update_runbook_entry",
  "action_id": "alex_053_update_runbook_entry",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_053_00"
}

2. tool_called / get_runbook

{
  "type": "tool_called",
  "tool": "get_runbook",
  "action_id": "alex_053_get_runbook",
  "check_id": "alex_053_01"
}

3. field_equals / update_runbook_entry

{
  "type": "field_equals",
  "path": "result.runbook.id",
  "value": "rb_shard_keeper_label_contract",
  "tool": "update_runbook_entry",
  "action_id": "alex_053_update_runbook_entry",
  "check_id": "alex_053_02"
}

4. field_llm_judge / update_runbook_entry

{
  "type": "field_llm_judge",
  "path": "result.runbook.body",
  "criterion": "The updated body says shard-keeper emits storage_region in production and rollup-service reads storage_region. It does not present shard_region as a current or alternative production key.",
  "tool": "update_runbook_entry",
  "action_id": "alex_053_update_runbook_entry",
  "check_id": "alex_053_03"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "update_runbook_entry",
        "action_id": "alex_053_update_runbook_entry",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_053_00"
      },
      {
        "type": "tool_called",
        "tool": "get_runbook",
        "action_id": "alex_053_get_runbook",
        "check_id": "alex_053_01"
      },
      {
        "type": "field_equals",
        "path": "result.runbook.id",
        "value": "rb_shard_keeper_label_contract",
        "tool": "update_runbook_entry",
        "action_id": "alex_053_update_runbook_entry",
        "check_id": "alex_053_02"
      },
      {
        "type": "field_llm_judge",
        "path": "result.runbook.body",
        "criterion": "The updated body says shard-keeper emits storage_region in production and rollup-service reads storage_region. It does not present shard_region as a current or alternative production key.",
        "tool": "update_runbook_entry",
        "action_id": "alex_053_update_runbook_entry",
        "check_id": "alex_053_03"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/053.yaml

SHA-256: d4e7c3c473940ed0313c633c68d9a44df4eca684184543b363c29558179fde14