DolphinBench

Test 122

Sep 14, 2026 / 1 fact

YAML

Request

Search my notes for Scaffold pricing-model material and create a scratchpad with the returned snippets and source labels.

Required memory

Fact 70

As of April 22, 2023, Morgan Chen requires anyone retrieving Morgan's notes from the vector store to use the `mchen-notes-v3` namespace; the old namespaces are stale and their results are not trusted.

Source evidence (1)

000425Apr 22, 2023 / 15:30 UTC-07:00

keep forgetting i renamed that. if anyone's pulling my notes from the vector store it has to be mchen-notes-v3. not the old namespaces -- those are stale and i do not trust what comes back from them anymore.

Message 000425 in history

Expected tool calls

  • embeddings_encode
  • vector_search
  • create_doc

Grading

1. field_equals / vector_search
{
  "type": "field_equals",
  "tool": "vector_search",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_122_00",
  "action_id": "morgan_122_vector_search"
}
2. field_equals_all_calls / vector_search
{
  "type": "field_equals_all_calls",
  "path": "args.namespace",
  "value": "mchen-notes-v3",
  "tool": "vector_search",
  "check_id": "morgan_122_01"
}
3. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_122_02",
  "action_id": "morgan_122_create_doc"
}
4. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "path": "result.document.body",
  "action_id": "morgan_122_create_doc",
  "check_id": "morgan_122_03",
  "criterion": "Contains the full returned snippet below, labeled with its document ID and mchen-notes-v3 as its source. Markdown quoting, headings, code fences, and whitespace changes are acceptable; do not require an exact layout. Repeating the same returned record is unnecessary. A summary replacing the snippet, missing material, invented material, or attributing it to old namespaces does not satisfy the request. A warning not to use old namespaces is valid. {\"id\": \"doc_1785274730003\", \"snippet\": \"Mercury pricing decision \\u2014 hybrid platform minimums\\n\\n# Mercury pricing decision \\u2014 hybrid platform minimums\\n\\n## Decision\\nWe are no longer treating Mercury pricing as a pure-MAU-versus-pure-seat debate. For new Mercury design-partner and post-launch quotes, use the hybrid model below.\\n\\n## Quote guidance\\n- Pilot platform minimum: $2,500/month, including up to 50 monthly active developers\\n- Growth platform minimum: $7,500/month, including up to 200 monthly active developers\\n- Overage: $1,000 per additional 50 monthly active developers\\n- Enterprise add-ons such as SSO, audit logs, and advanced admin controls are quoted separately only after those features actually ship\\n\\n## Rationale\\nThis keeps the usage-ramp logic honest while giving Devon a cleaner floor for forecasting and procurement conversations.\"}"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "vector_search",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_122_00",
        "action_id": "morgan_122_vector_search"
      },
      {
        "type": "field_equals_all_calls",
        "path": "args.namespace",
        "value": "mchen-notes-v3",
        "tool": "vector_search",
        "check_id": "morgan_122_01"
      },
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_122_02",
        "action_id": "morgan_122_create_doc"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "path": "result.document.body",
        "action_id": "morgan_122_create_doc",
        "check_id": "morgan_122_03",
        "criterion": "Contains the full returned snippet below, labeled with its document ID and mchen-notes-v3 as its source. Markdown quoting, headings, code fences, and whitespace changes are acceptable; do not require an exact layout. Repeating the same returned record is unnecessary. A summary replacing the snippet, missing material, invented material, or attributing it to old namespaces does not satisfy the request. A warning not to use old namespaces is valid. {\"id\": \"doc_1785274730003\", \"snippet\": \"Mercury pricing decision \\u2014 hybrid platform minimums\\n\\n# Mercury pricing decision \\u2014 hybrid platform minimums\\n\\n## Decision\\nWe are no longer treating Mercury pricing as a pure-MAU-versus-pure-seat debate. For new Mercury design-partner and post-launch quotes, use the hybrid model below.\\n\\n## Quote guidance\\n- Pilot platform minimum: $2,500/month, including up to 50 monthly active developers\\n- Growth platform minimum: $7,500/month, including up to 200 monthly active developers\\n- Overage: $1,000 per additional 50 monthly active developers\\n- Enterprise add-ons such as SSO, audit logs, and advanced admin controls are quoted separately only after those features actually ship\\n\\n## Rationale\\nThis keeps the usage-ramp logic honest while giving Devon a cleaner floor for forecasting and procurement conversations.\"}"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/122.yaml

SHA-256: c65c43cf63621231eec8668ae6418aa2ef2c0b4bde18cf26410432e06a79a785