DolphinBench

Test 080

Sep 14, 2026 / 1 fact

YAML

Request

Email sarah@scaffold.dev with the decision about whether Harbor Health's Mercury quote should use 180 purchased seats or 63 monthly active developers, and whether to add a one-off enterprise exception.

Required memory

Fact 138

Morgan Chen requested on August 31, 2023 that a short note be posted in #eng-team stating that Mercury pricing bands use monthly active developer usage rather than purchased seats or directory size, Scaffold/internal test users do not count, preview/sample-only activity is separate from activation reporting, Evergreen-facing answers must be budgetary and non-custom, and customer emails must not reopen the MAU-versus-seat debate or invent enterprise exceptions.

Source evidence (1)

000709Aug 31, 2023 / 10:06 UTC-07:00

Pricing/counting ambiguity is popping up after Sarah’s first budgetary wording. Discord — #eng-team Thread: Mercury pricing count / preview-only question Date: 2023-08-31 09:08 AM — Jake Quick pricing/counting question from the new Mercury budgetary wording. If an invited Evergreen user clicks in via magic link, looks around the preview/sample data, but never connects a real source or gets to a live sync, is that person part of the “monthly active developer” band for pricing, or not? Trying to make sure we don’t accidentally recreate seat logic by counting everyone who merely lands in the product. 09:12 AM — Anna Martinez I’d keep that separate. Preview/sample-only was deliberately non-activating in the external wave, and activation is still the corrected definition: real_source_connected or first_live_sync_completed within 7 days. If we let preview traffic do double duty as both “kind of active” and “kind of activated,” our reporting gets muddy fast. 09:14 AM — Jake Yep. I don’t want docs or dashboards to imply “opened preview once = active developer.” We also have internal Scaffold folks hitting the invite flow while testing org setup / retry paths. Those definitely shouldn’t roll into anything customer-facing. 09:17 AM — Anna Martinez Agreed on internal/test users staying out. Otherwise we’d be charging off our own dogfood and also polluting the early wave data. 09:21 AM — Sarah Kim I need a customer-safe sentence here because Evergreen may ask why the budgetary range is tied to monthly active developers instead of total seats in their directory. I can say “budget is based on actual monthly active developer usage in Mercury, not provisioned seats,” but I do not want to ad-lib what counts as active if the next question is “what about someone who only opened the preview?” 09:25 AM — Jake That’s exactly the edge case. Right now an invited person can successfully get into the bounded preview experience before anyone has connected a real source. That makes sense for product, but it leaves a counting gray area if someone reads “usage” too loosely. 09:29 AM — Devon Hayes The rule needs to stay forecastable. If this turns into a one-off exception list by customer, finance and budgeting conversations get messy immediately. Broadly: - pricing bands should map to monthly active developer usage, not directory size or purchased seats - internal Scaffold/test traffic should not count - preview/sample-only behavior should not get smuggled into activation reporting What I do not want is an Evergreen-specific carveout or a shadow seat count hiding underneath the usage model. 09:34 AM — Sarah Kim +1 on no Evergreen carveout. If they ask, I need to keep it budgetary and non-custom. I don’t want to imply a procurement packet, special enterprise policy, or roadmap trade in a one-line answer. 09:36 AM — Anna Martinez Also please don’t collapse “preview-only” into “active” just because we can see session events. We intentionally kept sample data preview-only and non-activating. Reporting-wise, that should remain a distinct bucket from actual activation. Otherwise every retention/activation chart gets re-litigated again. 09:41 AM — Jake Understood. My fear was that someone sees “Pilot includes up to 50 monthly active developers” and interprets that as “count every invited person who clicked once.” That gets us right back to seat-ish behavior, just with different words. 09:45 AM — Devon Hayes Right, and the model only works if the number is explainable without a long footnote. “Actual monthly active developer usage” is good. “Actual usage, except for these seven enterprise edge cases” is bad. Let’s not reopen the seat-versus-usage argument every time a broad directory exists. 09:49 AM — Sarah Kim For Evergreen-facing language, I’ll stay with budgetary guidance tied to actual monthly active developer usage and avoid promising how any future enterprise/security packaging would be counted. If they push past that, I’ll pause rather than improvise. 09:52 AM — Anna Martinez Thanks. And on the internal side, let’s keep preview/sample-only separate from the corrected activation definition in every doc/dashboard. Even if pricing uses an activity concept, activation still means real_source_connected or first_live_sync_completed within 7 days. 09:56 AM — Jake Makes sense. I’ll hold off on answering the eng docs question until we have one clean sentence everyone can reuse. Post a short channel note in #eng-team so this doesn’t stay buried in the thread. Clarify that Mercury pricing bands are based on monthly active developer usage, not purchased seats or directory size; Scaffold/internal test users don’t count; preview/sample-only activity stays separate from activation reporting; any Evergreen-facing answer stays budgetary and non-custom; and nobody should reopen the MAU-versus-seat debate or invent enterprise exceptions in customer emails.

Message 000709 in history

Expected tool calls

  • send_email

Grading

1. field_equals / send_email
{
  "type": "field_equals",
  "tool": "send_email",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_080_00",
  "action_id": "morgan_080_send_email"
}
2. field_equals / send_email
{
  "type": "field_equals",
  "path": "result.to",
  "value": "sarah@scaffold.dev",
  "tool": "send_email",
  "check_id": "morgan_080_01",
  "action_id": "morgan_080_send_email"
}
3. field_llm_judge / send_email
{
  "type": "field_llm_judge",
  "path": "result.body",
  "criterion": "Uses 63 monthly active developers rather than 180 purchased seats and rejects a one-off enterprise exception.",
  "tool": "send_email",
  "check_id": "morgan_080_02",
  "action_id": "morgan_080_send_email"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "send_email",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_080_00",
        "action_id": "morgan_080_send_email"
      },
      {
        "type": "field_equals",
        "path": "result.to",
        "value": "sarah@scaffold.dev",
        "tool": "send_email",
        "check_id": "morgan_080_01",
        "action_id": "morgan_080_send_email"
      },
      {
        "type": "field_llm_judge",
        "path": "result.body",
        "criterion": "Uses 63 monthly active developers rather than 180 purchased seats and rejects a one-off enterprise exception.",
        "tool": "send_email",
        "check_id": "morgan_080_02",
        "action_id": "morgan_080_send_email"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/080.yaml

SHA-256: aede2b8bd5fbf13049e3887060729be4f0e092ca471b57d76f13ea9a1cb6e4f6