Test 096

Jan 1, 2028 / 1 fact

Request

Inspect deploy-tools#940 and review its proposed laptop path for urgent production deploys and rollbacks.

Required memory

Fact 71

Alex Valdez and Hema settled that Wes must use the deploy pipeline rather than laptop deploys because laptop deploys bypass the canary.

Source evidence (1)

000386Aug 7, 2023 / 09:20 UTC-04:00

Wes asked before taking a small ingest-edge staging fix whether ingest-edge staging counts as canary-safe, or whether he's still limited to metrics-router non-prod under the May guidance. That exposed the annoying truth that the canary-safe environment guidance was still just a promised follow-up from the May shard-keeper laptop-deploy correction. Hema and I settled the practical rule: Wes has to use the deploy pipeline rather than laptop deploys because laptop deploys bypass the canary; he may operate solo in metrics-router non-prod and ingest-edge staging; shard-keeper stays outside his solo scope unless I or the Cyrus-team backup explicitly pairs with him. Create a short document titled Canary-safe environments — Wes deploy guidance in the Eng folder using the settled guidance below.

Message 000386 in history

Expected tool calls

get_pr
review_pr

Grading

1. field_equals / review_pr

{
  "type": "field_equals",
  "tool": "review_pr",
  "action_id": "alex_096_review_pr",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_096_00"
}

2. field_equals / get_pr

{
  "type": "field_equals",
  "tool": "get_pr",
  "action_id": "alex_096_read_pr",
  "path": "result.pr.id",
  "value": "deploy-tools#940",
  "check_id": "alex_096_01"
}

3. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.pr_id",
  "value": "deploy-tools#940",
  "tool": "review_pr",
  "action_id": "alex_096_review_pr",
  "check_id": "alex_096_02"
}

4. field_equals / review_pr

{
  "type": "field_equals",
  "path": "result.review.decision",
  "value": "request_changes",
  "tool": "review_pr",
  "action_id": "alex_096_review_pr",
  "check_id": "alex_096_03"
}

5. field_llm_judge / review_pr

{
  "type": "field_llm_judge",
  "path": "result.review.body",
  "criterion": "The review rejects the laptop or direct-deploy path because it bypasses the canary and requires Wes to use the deploy pipeline instead.",
  "tool": "review_pr",
  "action_id": "alex_096_review_pr",
  "check_id": "alex_096_04"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "review_pr",
        "action_id": "alex_096_review_pr",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_096_00"
      },
      {
        "type": "field_equals",
        "tool": "get_pr",
        "action_id": "alex_096_read_pr",
        "path": "result.pr.id",
        "value": "deploy-tools#940",
        "check_id": "alex_096_01"
      },
      {
        "type": "field_equals",
        "path": "result.review.pr_id",
        "value": "deploy-tools#940",
        "tool": "review_pr",
        "action_id": "alex_096_review_pr",
        "check_id": "alex_096_02"
      },
      {
        "type": "field_equals",
        "path": "result.review.decision",
        "value": "request_changes",
        "tool": "review_pr",
        "action_id": "alex_096_review_pr",
        "check_id": "alex_096_03"
      },
      {
        "type": "field_llm_judge",
        "path": "result.review.body",
        "criterion": "The review rejects the laptop or direct-deploy path because it bypasses the canary and requires Wes to use the deploy pipeline instead.",
        "tool": "review_pr",
        "action_id": "alex_096_review_pr",
        "check_id": "alex_096_04"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/alex/096.yaml

SHA-256: af6629bb54508978709c28a243a97ffc026f75cafe70cecf7facc32437419171