DolphinBench

Test 093

Sep 14, 2026 / 1 fact

YAML

Request

Post a review comment on scaffold/integrations#4120 correcting the current API and authentication guidance pasted from an old integration guide.

Required memory

Fact 58

By 2023-04-14, Morgan Chen reported that the API v2 migration was complete: the API used GraphQL with JWT authentication, all clients had cut over cleanly, and API v1 was shut down on the morning of 2023-04-14.

Source evidence (1)

000365Apr 14, 2023 / 13:00 UTC-07:00

v2 finally all the way over. api is GraphQL now, JWT auth, whole thing. killed off v1 this morning. all clients cut clean which honestly i kept expecting not to happen. idk. big one. then got on BART and immediately had to think about whether i left Kibo's food bin open

Message 000365 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_093_00",
  "action_id": "morgan_093_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/integrations#4120",
  "tool": "post_pr_comment",
  "check_id": "morgan_093_01",
  "action_id": "morgan_093_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Directs implementers to the current GraphQL API with JWT authentication and identifies v1 as retired or legacy, not an active integration path. Saying current GraphQL API with JWT and retired v1 is sufficient without spelling out v2. Recommending v1 or a different API/authentication method fails.",
  "tool": "post_pr_comment",
  "check_id": "morgan_093_02",
  "action_id": "morgan_093_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_093_00",
        "action_id": "morgan_093_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/integrations#4120",
        "tool": "post_pr_comment",
        "check_id": "morgan_093_01",
        "action_id": "morgan_093_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Directs implementers to the current GraphQL API with JWT authentication and identifies v1 as retired or legacy, not an active integration path. Saying current GraphQL API with JWT and retired v1 is sufficient without spelling out v2. Recommending v1 or a different API/authentication method fails.",
        "tool": "post_pr_comment",
        "check_id": "morgan_093_02",
        "action_id": "morgan_093_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/093.yaml

SHA-256: 5fb0e31e3609882ab9b49335c65881e30cddb961f9b19f803cc7e665e9ee8d19