DolphinBench

Test 059

Sep 14, 2026 / 1 fact

YAML

Request

PR scaffold/retrieval#2407 switches my notes lookup to mchen-notes-v2. Post a review comment saying whether that namespace is correct and what to use if it is not.

Required memory

Fact 70

As of April 22, 2023, Morgan Chen requires anyone retrieving Morgan's notes from the vector store to use the `mchen-notes-v3` namespace; the old namespaces are stale and their results are not trusted.

Source evidence (1)

000425Apr 22, 2023 / 15:30 UTC-07:00

keep forgetting i renamed that. if anyone's pulling my notes from the vector store it has to be mchen-notes-v3. not the old namespaces -- those are stale and i do not trust what comes back from them anymore.

Message 000425 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_059_00",
  "action_id": "morgan_059_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/retrieval#2407",
  "tool": "post_pr_comment",
  "check_id": "morgan_059_01",
  "action_id": "morgan_059_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Requires mchen-notes-v3 instead of mchen-notes-v2 because the older namespace is stale or untrusted. Merely mentioning v3 without selecting it fails.",
  "tool": "post_pr_comment",
  "check_id": "morgan_059_02",
  "action_id": "morgan_059_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_059_00",
        "action_id": "morgan_059_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/retrieval#2407",
        "tool": "post_pr_comment",
        "check_id": "morgan_059_01",
        "action_id": "morgan_059_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Requires mchen-notes-v3 instead of mchen-notes-v2 because the older namespace is stale or untrusted. Merely mentioning v3 without selecting it fails.",
        "tool": "post_pr_comment",
        "check_id": "morgan_059_02",
        "action_id": "morgan_059_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/059.yaml

SHA-256: 87af1beabb4222947ec22eda1e5efc4da0dec52e25cd46c26ef2ff44de2480c7