Test 150

Jan 1, 2028 / 2 facts

Request

Inspect infra-history#1432 and comment with shard-keeper's place and status in the March 7, 2023 migration decision.

Required memory

Fact 3

The metrics pipeline migration cutover order is now fixed as ingest-edge first because it has the lowest blast radius, metrics-router second, and shard-keeper last because its long-tail cardinality logic gets complicated.

Source evidence (1)

000007Mar 7, 2023 / 10:00 UTC-05:00

just got off w Hema + Nadia. order is fixed now: ingest-edge first -- lowest blast radius, then metrics-router, shard-keeper last bc that's where the long-tail cardinality logic gets weird. Hema's basically tracking it at the manager level, not trying to run it day to day. Nadia had one real ask: keep label names parity on the dashboards during cutover so alert defs don't turn into a cleanup project at the same time. fine w that. shard-keeper itself hasn't started yet anyway, which honestly makes the ordering easier

Message 000007 in history

Fact 4

As of March 7, 2023, work on shard-keeper for the metrics pipeline migration has not started.

Source evidence (1)

000007Mar 7, 2023 / 10:00 UTC-05:00

Message 000007 in history

Expected tool calls

get_pr
post_pr_comment

Grading

1. field_equals / post_pr_comment

{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "action_id": "alex_150_post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_150_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_150_read_pr",
  "path": "result.pr.id",
  "value": "infra-history#1432",
  "check_id": "alex_150_01"
}

3. field_equals / post_pr_comment

{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "infra-history#1432",
  "tool": "post_pr_comment",
  "action_id": "alex_150_post_pr_comment",
  "check_id": "alex_150_02"
}

4. field_llm_judge / post_pr_comment

{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "The comment says the March 7 order was ingest-edge first, metrics-router second, and shard-keeper deliberately last, and that shard-keeper work had not started at that checkpoint. It does not claim what happened later.",
  "tool": "post_pr_comment",
  "action_id": "alex_150_post_pr_comment",
  "check_id": "alex_150_03"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "action_id": "alex_150_post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_150_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_150_read_pr",
        "path": "result.pr.id",
        "value": "infra-history#1432",
        "check_id": "alex_150_01"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "infra-history#1432",
        "tool": "post_pr_comment",
        "action_id": "alex_150_post_pr_comment",
        "check_id": "alex_150_02"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "The comment says the March 7 order was ingest-edge first, metrics-router second, and shard-keeper deliberately last, and that shard-keeper work had not started at that checkpoint. It does not claim what happened later.",
        "tool": "post_pr_comment",
        "action_id": "alex_150_post_pr_comment",
        "check_id": "alex_150_03"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/150.yaml

SHA-256: ce6af84e676b6c3656baebe6365787731695d5f14a6020b5a9235443b444d1fe