Test 180

Jan 1, 2028 / 1 fact

Request

Inspect otel-schema#590 and review its production deployment.environment behavior across the metrics path. State the allowed and rejected values, missing-input behavior, per-tenant limit, and every production component that must show parity.

Required memory

Fact 121

Alex Valdez reports that the forty-eight-hour production observation of the `deployment.environment` rollout is complete: across production OTel ingestion, metrics-router, and rollup-service, mixed-case allowed inputs normalize only to `prod`, `staging`, or `dev`; missing input omits the label; unknown, `qa`, malformed, and arbitrary raw values are rejected before publication and create no series; no tenant has more than three distinct values; and rollup-service reports no producer-consumer mismatch.

Source evidence (1)

001752Aug 8, 2024 / 09:20 UTC-04:00

The forty-eight-hour production observation is complete. Across the production OTel ingestion, metrics-router, and rollup-service path, mixed-case allowed input normalizes only to `prod`, `staging`, or `dev`; missing input omits the label; and unknown, `qa`, malformed, and arbitrary raw values are rejected before publication and create no series. No tenant exceeds three distinct `deployment.environment` values, and rollup-service reports no producer-consumer mismatch. The earlier `unset` legend is only a dashboard rendering of an absent key, not a published value. The mapped service owners completed the rollout, and production availability is recorded without transferring implementation or operational ownership to me.

Message 001752 in history

Expected tool calls

get_pr
review_pr

Grading

1. field_equals / review_pr

{
  "type": "field_equals",
  "tool": "review_pr",
  "action_id": "alex_180_review_pr",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_180_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_180_read_pr",
  "path": "result.pr.id",
  "value": "otel-schema#590",
  "check_id": "alex_180_01"
}

3. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.pr_id",
  "value": "otel-schema#590",
  "tool": "review_pr",
  "action_id": "alex_180_review_pr",
  "check_id": "alex_180_02"
}

4. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.decision",
  "value": "request_changes",
  "tool": "review_pr",
  "action_id": "alex_180_review_pr",
  "check_id": "alex_180_03"
}

5. field_llm_judge / review_pr

{
  "type": "field_llm_judge",
  "path": "result.review.body",
  "criterion": "The review requires mixed-case allowed inputs to normalize only to prod, staging, or dev; missing input to omit the label; unknown, qa, malformed, and arbitrary raw values to be rejected before publication without creating a series; no more than three distinct values per tenant; and parity through production OTel ingestion, metrics-router, and rollup-service.",
  "tool": "review_pr",
  "action_id": "alex_180_review_pr",
  "check_id": "alex_180_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "review_pr",
        "action_id": "alex_180_review_pr",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_180_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_180_read_pr",
        "path": "result.pr.id",
        "value": "otel-schema#590",
        "check_id": "alex_180_01"
      },
      {
        "type": "field_equals",
        "path": "result.review.pr_id",
        "value": "otel-schema#590",
        "tool": "review_pr",
        "action_id": "alex_180_review_pr",
        "check_id": "alex_180_02"
      },
      {
        "type": "field_equals",
        "path": "result.review.decision",
        "value": "request_changes",
        "tool": "review_pr",
        "action_id": "alex_180_review_pr",
        "check_id": "alex_180_03"
      },
      {
        "type": "field_llm_judge",
        "path": "result.review.body",
        "criterion": "The review requires mixed-case allowed inputs to normalize only to prod, staging, or dev; missing input to omit the label; unknown, qa, malformed, and arbitrary raw values to be rejected before publication without creating a series; no more than three distinct values per tenant; and parity through production OTel ingestion, metrics-router, and rollup-service.",
        "tool": "review_pr",
        "action_id": "alex_180_review_pr",
        "check_id": "alex_180_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/180.yaml

SHA-256: 310e1a6c6788d533bb98e784a90106a8768872e7e872c6c1d43bdc8a2cf9c5a2