DolphinBench

Test 197

Sep 14, 2026 / 1 fact

YAML

Request

Create an archive snippet comparing Clerk, Auth0, and custom auth. Use this brief: Clerk supplies hosted email/password, magic links, sessions, and organizations quickly but adds a dependency. Auth0 is hosted auth that avoids an in-house build, but pricing required repeated sales conversations. Custom auth offers control over organizations, invites, and administration, but costs Devon and Rishi two to three weeks away from agent work.

Required memory

Fact 47

Morgan Chen requested a skimmable auth comparison of Clerk, Auth0, and custom auth, with exactly three bullets per option covering what it is, its upsides, and its biggest worry, rather than a memo.

Source evidence (1)

000320Apr 9, 2023 / 16:00 UTC-07:00

need the auth thing in a format i can skim. Clerk, Auth0, custom auth. 3 bullets each only: what it is, upsides, biggest worry. not a memo.

Message 000320 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_197_00",
  "action_id": "morgan_197_create_doc"
}
2. field_markdown_bullets / create_doc
{
  "type": "field_markdown_bullets",
  "path": "result.document.body",
  "value": {
    "exact_items": 9
  },
  "tool": "create_doc",
  "check_id": "morgan_197_01",
  "action_id": "morgan_197_create_doc"
}
3. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Uses exactly three substantive bullets for each of Clerk, Auth0, and custom auth: what it is, upsides, and biggest worry. Allows short option labels but no extra comparison table, memo, or recommendation outside those bullets.",
  "tool": "create_doc",
  "check_id": "morgan_197_02",
  "action_id": "morgan_197_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_197_00",
        "action_id": "morgan_197_create_doc"
      },
      {
        "type": "field_markdown_bullets",
        "path": "result.document.body",
        "value": {
          "exact_items": 9
        },
        "tool": "create_doc",
        "check_id": "morgan_197_01",
        "action_id": "morgan_197_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Uses exactly three substantive bullets for each of Clerk, Auth0, and custom auth: what it is, upsides, and biggest worry. Allows short option labels but no extra comparison table, memo, or recommendation outside those bullets.",
        "tool": "create_doc",
        "check_id": "morgan_197_02",
        "action_id": "morgan_197_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/197.yaml

SHA-256: 5450417e20299c3f72d323b7c5bb686928978b4b3fd3cae60e695a6a6fee199d