Test 045

Jan 1, 2028 / 2 facts

Request

Check metrics-router p99_ms and replay-validator mismatch_rate for the last hour, then post a recommendation in #incident-metrics. Rollback and restoring legacy-aggregator have been proposed.

Required memory

Fact 81

The metrics-router dashboard legend now says rollback discussion should start from live-path signals, while validation-source movement is evidence for investigation rather than rollback criteria.

Source evidence (1)

000595Oct 9, 2023 / 09:20 UTC-04:00

Wes revised the dashboard-label cleanup after I pushed back on `validation-only, safe to ignore`. The new labels split live metrics-router traffic and live canary hosts from replay and mirror validation samples, and the legend now says rollback discussion should start from live-path signals while validation-source movement is investigation evidence rather than rollback criteria. I approved that direction because it keeps the distinction between rollback criteria and evidence for mismatch investigation or the Q4 controls work.

Message 000595 in history

Fact 99

Legacy-aggregator mirrored writes have been stopped; the service is now archived legacy code and is no longer an active replay source.

Source evidence (1)

000871Dec 18, 2023 / 10:22 UTC-05:00

The final legacy-aggregator stop gate passed. Roman's final replay fixture pass came back clean, live metrics-router and rollup-service references are absent, Nadia confirmed the relevant Cardinality Guardrails coverage for the prior label-change class, and Cyrus agreed the remaining validation should run through the OTel/Prometheus path plus Guardrails fixtures rather than legacy mirrored writes. We stopped legacy-aggregator's mirrored writes, so it is archived legacy code now — not a live path, not a production rollback path, and not an active replay source. Please update runbook entry `rb_legacy_aggregator_retirement_exit_gate` with the post-stop body below and keep the title `legacy-aggregator: mirrored-write retirement exit gate`.

Message 000871 in history

Expected tool calls

get_service_metrics
send_slack_message

Grading

1. field_equals / send_slack_message

{
  "type": "field_equals",
  "tool": "send_slack_message",
  "action_id": "alex_045_send_slack_message",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_045_00"
}

2. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_live_metric",
  "path": "result.service",
  "value": "metrics-router",
  "check_id": "alex_045_01"
}

3. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_live_metric",
  "path": "result.metric",
  "value": "p99_ms",
  "check_id": "alex_045_02"
}

4. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_live_metric",
  "path": "result.timerange",
  "value": "1h",
  "check_id": "alex_045_03"
}

5. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_validation_metric",
  "path": "result.service",
  "value": "replay-validator",
  "check_id": "alex_045_04"
}

6. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_validation_metric",
  "path": "result.metric",
  "value": "mismatch_rate",
  "check_id": "alex_045_05"
}

7. field_equals / get_service_metrics

{
  "type": "field_equals",
  "tool": "get_service_metrics",
  "action_id": "alex_045_validation_metric",
  "path": "result.timerange",
  "value": "1h",
  "check_id": "alex_045_06"
}

8. field_equals / send_slack_message

{
  "type": "field_equals",
  "path": "result.channel",
  "value": "#incident-metrics",
  "tool": "send_slack_message",
  "action_id": "alex_045_send_slack_message",
  "check_id": "alex_045_07"
}

9. field_llm_judge / send_slack_message

{
  "type": "field_llm_judge",
  "path": "result.message",
  "criterion": "The post recognizes that live metrics-router p99 is steady while replay-validator mismatch rate is rising. It says validation-only movement is investigation evidence rather than rollback criteria, recommends investigating the mismatch instead of rolling back, and rejects restoring legacy-aggregator because it is archived rather than an active replay or production service.",
  "tool": "send_slack_message",
  "action_id": "alex_045_send_slack_message",
  "check_id": "alex_045_08"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_slack_message",
        "action_id": "alex_045_send_slack_message",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_045_00"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_live_metric",
        "path": "result.service",
        "value": "metrics-router",
        "check_id": "alex_045_01"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_live_metric",
        "path": "result.metric",
        "value": "p99_ms",
        "check_id": "alex_045_02"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_live_metric",
        "path": "result.timerange",
        "value": "1h",
        "check_id": "alex_045_03"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_validation_metric",
        "path": "result.service",
        "value": "replay-validator",
        "check_id": "alex_045_04"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_validation_metric",
        "path": "result.metric",
        "value": "mismatch_rate",
        "check_id": "alex_045_05"
      },
      {
        "type": "field_equals",
        "tool": "get_service_metrics",
        "action_id": "alex_045_validation_metric",
        "path": "result.timerange",
        "value": "1h",
        "check_id": "alex_045_06"
      },
      {
        "type": "field_equals",
        "path": "result.channel",
        "value": "#incident-metrics",
        "tool": "send_slack_message",
        "action_id": "alex_045_send_slack_message",
        "check_id": "alex_045_07"
      },
      {
        "type": "field_llm_judge",
        "path": "result.message",
        "criterion": "The post recognizes that live metrics-router p99 is steady while replay-validator mismatch rate is rising. It says validation-only movement is investigation evidence rather than rollback criteria, recommends investigating the mismatch instead of rolling back, and rejects restoring legacy-aggregator because it is archived rather than an active replay or production service.",
        "tool": "send_slack_message",
        "action_id": "alex_045_send_slack_message",
        "check_id": "alex_045_08"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/045.yaml

SHA-256: 2770e0370d9e5cbcc3928f0c0be5e4e94e779dfbd77107f08f05dc1e4c757556