DolphinBench
DiscordGitHub

Test 017

Sep 14, 2026 / 1 fact

YAML

Request

Create a reusable customer-support macro for API v2 JWT nbf errors during token setup.

Required memory

Fact 289

By April 2, 2026, Leo had shipped the approved Atlas/API v2 customer-docs patch following repeated JWT `nbf` clock-skew support cases; the live guidance now explicitly says to allow up to five minutes of clock skew when setting `nbf` and not to copy sample timestamps directly into production tokens.

Source evidence (1)

002847Apr 2, 2026 / 08:44 UTC-07:00

Just logging that Leo shipped the small Atlas/API v2 docs patch we approved after the repeated JWT `nbf` clock-skew support cases. The live customer guidance now explicitly says to allow up to five minutes of clock skew when setting `nbf` and not to copy sample timestamps directly into production tokens. Sarah is keeping the handling on the normal docs/support route. There is still no Atlas product-priority item unless Leo finds a real verification-code defect.

Message 002847 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_017_00",
  "action_id": "morgan_017_create_doc"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "Explains the allowance of up to five minutes (300 seconds) of clock skew for JWT nbf values and warns against copying sample timestamps into production tokens. Does not suggest increasing the tolerance beyond five minutes.",
  "tool": "create_doc",
  "check_id": "morgan_017_01",
  "action_id": "morgan_017_create_doc"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_017_00",
        "action_id": "morgan_017_create_doc"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "Explains the allowance of up to five minutes (300 seconds) of clock skew for JWT nbf values and warns against copying sample timestamps into production tokens. Does not suggest increasing the tolerance beyond five minutes.",
        "tool": "create_doc",
        "check_id": "morgan_017_01",
        "action_id": "morgan_017_create_doc"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/017.yaml

SHA-256: bd980cf3502b0768f11c22fc10b46328b280bb0d635109bb3be553c43dfc7358