Test 042

Sep 14, 2026 / 1 fact

Request

Email support@scaffold.dev a customer-ready explanation of API v2 webhook delivery behavior and how their integration should process incoming events.

Required memory

Fact 316

The completed internal examples cover at-least-once webhook retries and idempotency by event ID, cursor pagination versus fresh polling, and JWT five-minute clock-skew leeway.

Source evidence (1)

003271Aug 4, 2026 / 10:36 UTC-07:00

Leo and Sarah finished the Atlas/API v2 support-contract examples in the narrow form I wanted. The internal examples now cover at-least-once webhook retries and idempotency by event ID, cursor pagination versus fresh polling, and JWT five-minute clock-skew leeway. Leo kept each example tied to logs and request IDs: webhook retries use event and delivery IDs rather than exactly-once language, cursor guidance uses durable checkpoints and request IDs rather than fresh polling as a default, and JWT guidance stays at five minutes with JWKS cache freshness checks before asking for more logs. Support now has stable first-pass answers, and this does not create hidden reliability-roadmap work or a Jake product-priority review lane unless logs show a real product-priority or sequencing issue.

Message 003271 in history

Expected tool calls

send_email

Grading

1. field_equals / send_email

{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_042_00",
  "action_id": "morgan_042_send_email"
}

2. field_equals / send_email

{
  "type": "field_equals",
  "path": "result.to",
  "value": "support@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_042_01",
  "action_id": "morgan_042_send_email"
}

3. field_llm_judge / send_email

{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Explains at-least-once webhook retries rather than an exactly-once guarantee, and tells the customer to handle deliveries idempotently by event ID.",
  "tool": "send_email",
  "check_id": "morgan_042_02",
  "action_id": "morgan_042_send_email"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_042_00",
        "action_id": "morgan_042_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "support@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_042_01",
        "action_id": "morgan_042_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Explains at-least-once webhook retries rather than an exactly-once guarantee, and tells the customer to handle deliveries idempotently by event ID.",
        "tool": "send_email",
        "check_id": "morgan_042_02",
        "action_id": "morgan_042_send_email"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/morgan/042.yaml

SHA-256: 1b09d5061a9b31f77dc7404c4dc48187ed9dcf74c85dd7bdb765e7ff7597e84c