Test 134

Jan 1, 2028 / 1 fact

Request

Inspect shard-keeper#1884 and review its production region-label behavior.

Required memory

Fact 108

Roman completed the production rollout of the shard-keeper label rename: shard-keeper now emits `storage_region`, rollup-service reads it, and one hour of production observation showed matching label counts, zero dropped rollups, normal rollup completion, and no change to alert semantics.

Source evidence (1)

001139Feb 20, 2024 / 15:50 UTC-05:00

Roman finished the production rollout. Shard-keeper now emits `storage_region`, rollup-service reads that key, and an hour of production observation shows matching label counts, zero dropped rollups, normal rollup completion, and no alert-semantic change. The shared fixture still contains the executable old-to-new parity case that caught the original mismatch, so the blocked rename has now landed safely.

Message 001139 in history

Expected tool calls

get_pr
review_pr

Grading

1. field_equals / review_pr

{
  "type": "field_equals",
  "tool": "review_pr",
  "action_id": "alex_134_review_pr",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_134_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_134_read_pr",
  "path": "result.pr.id",
  "value": "shard-keeper#1884",
  "check_id": "alex_134_01"
}

3. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.pr_id",
  "value": "shard-keeper#1884",
  "tool": "review_pr",
  "action_id": "alex_134_review_pr",
  "check_id": "alex_134_02"
}

4. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.decision",
  "value": "request_changes",
  "tool": "review_pr",
  "action_id": "alex_134_review_pr",
  "check_id": "alex_134_03"
}

5. field_llm_judge / review_pr

{
  "type": "field_llm_judge",
  "path": "result.review.body",
  "criterion": "The review says storage_region is the production label emitted by shard-keeper and read by rollup-service, so the PR has reversed the current and fallback keys and must be corrected.",
  "tool": "review_pr",
  "action_id": "alex_134_review_pr",
  "check_id": "alex_134_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "review_pr",
        "action_id": "alex_134_review_pr",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_134_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_134_read_pr",
        "path": "result.pr.id",
        "value": "shard-keeper#1884",
        "check_id": "alex_134_01"
      },
      {
        "type": "field_equals",
        "path": "result.review.pr_id",
        "value": "shard-keeper#1884",
        "tool": "review_pr",
        "action_id": "alex_134_review_pr",
        "check_id": "alex_134_02"
      },
      {
        "type": "field_equals",
        "path": "result.review.decision",
        "value": "request_changes",
        "tool": "review_pr",
        "action_id": "alex_134_review_pr",
        "check_id": "alex_134_03"
      },
      {
        "type": "field_llm_judge",
        "path": "result.review.body",
        "criterion": "The review says storage_region is the production label emitted by shard-keeper and read by rollup-service, so the PR has reversed the current and fallback keys and must be corrected.",
        "tool": "review_pr",
        "action_id": "alex_134_review_pr",
        "check_id": "alex_134_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/134.yaml

SHA-256: 31b843f30998cec841b43e7068839af27dcd38ae75a515fc17fab23fb7100c02