Test 019

Sep 14, 2026 / 1 fact

YAML

Request

Post a review comment on scaffold/web#1842 telling the author which indentation style to use before merge.

Required memory

Fact 1

Morgan Chen prefers tabs rather than spaces.

Source evidence (1)

000001Jan 9, 2023 / 08:00 UTC-08:00

ok so. morgan chen. Scaffold is the company Devon and I started -- developer data plumbing, basically. customers create a workspace, connect their data sources, keep them synced, then use the data through our APIs and connectors. mission apartment, kibo shedding on everything. idk, useful to say it plainly once. 2 years in, still tabs not spaces, still doing blue bottle if i'm out early enough. jamie's on evening shifts again this month so the apartment's been weirdly quiet.

Message 000001 in history

Expected tool calls

post_pr_comment

Grading

1. field_equals / post_pr_comment

{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_019_00",
  "action_id": "morgan_019_post_pr_comment"
}

2. field_equals / post_pr_comment

{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/web#1842",
  "tool": "post_pr_comment",
  "check_id": "morgan_019_01",
  "action_id": "morgan_019_post_pr_comment"
}

3. field_llm_judge / post_pr_comment

{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Tells the author to use tabs rather than spaces for indentation. Quoting tabs only to reject them fails.",
  "tool": "post_pr_comment",
  "check_id": "morgan_019_02",
  "action_id": "morgan_019_post_pr_comment"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_019_00",
        "action_id": "morgan_019_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/web#1842",
        "tool": "post_pr_comment",
        "check_id": "morgan_019_01",
        "action_id": "morgan_019_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Tells the author to use tabs rather than spaces for indentation. Quoting tabs only to reject them fails.",
        "tool": "post_pr_comment",
        "check_id": "morgan_019_02",
        "action_id": "morgan_019_post_pr_comment"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/morgan/019.yaml

SHA-256: cae0659d7e7bbace7aadec2eeb96c1944c2d2acf8d8c0efe552e3fe5705febdc