Test 076 for Alex Valdez

Request

Inspect incident-archive#418 and correct which of rollup-service and metric-rollup missed the shard-keeper label change, then explain what the other service is.

Required memory

Fact 30

Alex Valdez believed INC-2023-04-18-001 was caused by an earlier label-name change reaching shard-keeper but not rollup-service, which triggered all symptom alerts and no cause alerts.

Source evidence (1)

000103Apr 18, 2023 / 23:45 UTC-04:00

9:42 shard-keeper page and i'm still in it. think i finally have the stupid diff -- label-name change from earlier made it into shard-keeper, never got picked up by rollup-service, so all the symptom alerts went off and none of the cause ones did. INC-2023-04-18-001 already has my name all over it i guess. haven't paged Hema yet. if this rollback behaves maybe 20 min, but the path is fragile and i do not want to widen this at 11:45 for no reason. looks like sev2, probably lands around ~5h45m by the time it's actually clean. impaired ingestion basically the whole window. i don't think we lost data. just... great.

Message 000103 in history

Fact 53

Alex Valdez says `metric-rollup` is deprecated and must not be touched.

Source evidence (1)

000186May 23, 2023 / 14:00 UTC-04:00

Need Wes anchored on this bc the names are close and people keep collapsing them. As of May 23: `metric-rollup` = the old deprecated thing I inherited, basically un-owned, do not touch. `rollup-service` = different service, Cyrus-team-owned after the audit, and that's the one the Apr 18 outage hit. Same prefix, different code paths, different owners. if someone says "rollup" in #infra without the exact name, I'd bounce it back.

Message 000186 in history

Fact 54

Alex Valdez clarifies that `rollup-service`, not `metric-rollup`, was affected by the April 18 outage; the similarly named services have different code paths.

Source evidence (1)

000186May 23, 2023 / 14:00 UTC-04:00

Need Wes anchored on this bc the names are close and people keep collapsing them. As of May 23: `metric-rollup` = the old deprecated thing I inherited, basically un-owned, do not touch. `rollup-service` = different service, Cyrus-team-owned after the audit, and that's the one the Apr 18 outage hit. Same prefix, different code paths, different owners. if someone says "rollup" in #infra without the exact name, I'd bounce it back.

Message 000186 in history

Fact 55

Alex Valdez wants any use of the ambiguous name "rollup" in #infra returned for clarification unless the exact service name is provided.

Source evidence (1)

000186May 23, 2023 / 14:00 UTC-04:00

Need Wes anchored on this bc the names are close and people keep collapsing them. As of May 23: `metric-rollup` = the old deprecated thing I inherited, basically un-owned, do not touch. `rollup-service` = different service, Cyrus-team-owned after the audit, and that's the one the Apr 18 outage hit. Same prefix, different code paths, different owners. if someone says "rollup" in #infra without the exact name, I'd bounce it back.

Message 000186 in history

Expected tool calls

get_pr
review_pr

Grading

1. field_equals / review_pr

{
  "type": "field_equals",
  "tool": "review_pr",
  "action_id": "alex_076_review_pr",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_076_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_076_read_pr",
  "path": "result.pr.id",
  "value": "incident-archive#418",
  "check_id": "alex_076_01"
}

3. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.pr_id",
  "value": "incident-archive#418",
  "tool": "review_pr",
  "action_id": "alex_076_review_pr",
  "check_id": "alex_076_02"
}

4. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.decision",
  "value": "request_changes",
  "tool": "review_pr",
  "action_id": "alex_076_review_pr",
  "check_id": "alex_076_03"
}

5. field_llm_judge / review_pr

{
  "type": "field_llm_judge",
  "path": "result.review.body",
  "criterion": "The review identifies rollup-service as the downstream consumer that missed the shard-keeper label change. It clearly identifies metric-rollup as a different, deprecated service and says the archive must use exact service names.",
  "tool": "review_pr",
  "action_id": "alex_076_review_pr",
  "check_id": "alex_076_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "review_pr",
        "action_id": "alex_076_review_pr",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_076_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_076_read_pr",
        "path": "result.pr.id",
        "value": "incident-archive#418",
        "check_id": "alex_076_01"
      },
      {
        "type": "field_equals",
        "path": "result.review.pr_id",
        "value": "incident-archive#418",
        "tool": "review_pr",
        "action_id": "alex_076_review_pr",
        "check_id": "alex_076_02"
      },
      {
        "type": "field_equals",
        "path": "result.review.decision",
        "value": "request_changes",
        "tool": "review_pr",
        "action_id": "alex_076_review_pr",
        "check_id": "alex_076_03"
      },
      {
        "type": "field_llm_judge",
        "path": "result.review.body",
        "criterion": "The review identifies rollup-service as the downstream consumer that missed the shard-keeper label change. It clearly identifies metric-rollup as a different, deprecated service and says the archive must use exact service names.",
        "tool": "review_pr",
        "action_id": "alex_076_review_pr",
        "check_id": "alex_076_04"
      }
    ]
  }
}