DolphinBench

Test 190

Sep 14, 2026 / 4 facts

YAML

Request

On scaffold/compass-archive#2718, correct what the April 2025 source-recency review permitted for Compass owner prompts and wider exposure.

Required memory

Fact 225

Anna assessed the Project Compass April 2025 prompts as usable for an operating review if each carries its definition, evidence source, date, and caveat, but not as clean customer-facing proof.

Source evidence (1)

001585Apr 22, 2025 / 16:47 UTC-07:00

Anna sent her Compass evidence pass. I want this turned into a short addendum for Friday, with the definitions/source/date/caveat attached so the prompts don't read like clean product metrics. Forwarding her table: Renewal-risk / admin-friction prompt - Action trigger: account-owner follow-up when an account has renewal-risk context plus documented admin friction. - Evidence has to include the account-thread source, date, owner, and caveat. - Evidence sources here: Sarah's Evergreen annual-account notes and Acme live-only notes. - Caveats: single-account notes are not repeatability proof; stale account-source timing has to be called out; this should not become a score or dashboard. Post-first-live-sync expansion prompt - Action trigger: an account-owner expansion-prep question only after first live sync is actually live and teammate-invite/admin-role basics are visible. - Evidence sources: activation events and account-thread notes. - Caveats: sample/preview behavior is excluded; source timing travels with the prompt; this does not imply a customer is ready to expand. Anna's read: usable for an operating review if each prompt carries definition, evidence source, date, and caveat. Not clean customer-facing proof. Make a Compass April evidence-matrix addendum in bullets with each prompt's action trigger, evidence source, and caveats.

Message 001585 in history

Fact 226

Two Acme notes in the April 23, 2025 Compass evidence are too thin because their owner/source and date are unclear; Sarah Kim said to exclude or caveat them.

Source evidence (1)

001586Apr 23, 2025 / 09:34 UTC-07:00

Sarah sent the account-thread evidence. I want a separate addendum that keeps the Evergreen/Acme boundary clean and marks the thin notes as thin. Forwarding Sarah's notes: Evergreen Bank - Account-thread evidence is annual-account current-state only. - Safe evidence includes the annual Mercury Growth account context, shipped SAML rotation path, visible invite/source/admin audit lookup support, workspace-admin support route, and the recently verified basic invited-teammate/source-owner cues in the current activation path. - Caveat: clean evidence, but still a single annual account. Do not turn it into repeatable customer-growth proof. Acme - Evidence stays live-only review of Acme's own account-health, reliability, and renewal-risk signals. - No Evergreen-derived examples, screenshots, written Mercury packets, or customer-specific briefing materials should go out. - Two Acme notes are too thin because the owner/source and date are unclear; exclude them or caveat them. - The live review produced clarification questions for the account owner, not customer action. Sarah's read: the prompt work is improving internal account-thread hygiene and prep questions, not yet changing customer behavior. Create an account-thread addendum for the Compass review that separates Evergreen, Acme, excluded/thin evidence, and the behavior-change read.

Message 001586 in history

Fact 227

Leo’s final seam note for the April 25 prompt review reported that the renewal-risk/admin-friction prompt’s source-recency handling is now adequate for continued owner-group use.

Source evidence (1)

001589Apr 24, 2025 / 15:37 UTC-07:00

Leo sent the last seam note for Friday. I want this phrased as a clean risk/evidence line, not proof language. Forwarding Leo: - The renewal-risk/admin-friction prompt now carries source-recency handling well enough for continued owner-group use. - When account-thread timing is close or source ownership is unclear, the prompt should show the evidence date and caveat. - The post-first-live-sync prompt is safe for owner-group review when the first_live_sync_completed event is present and sample/preview behavior is excluded. - Caveat: this does not justify broader exposure, customer-facing claims, or a dashboard-style surface. - Jake should keep implementation prompt-level unless the review explicitly changes scope. Draft the Leo section as a concise risk/evidence line for the Apr 25 review.

Message 001589 in history

Fact 228

Leo’s final seam note for the April 25 prompt review said the renewal-risk/admin-friction and post-first-live-sync prompts do not justify a dashboard-style surface.

Source evidence (1)

001589Apr 24, 2025 / 15:37 UTC-07:00

Leo sent the last seam note for Friday. I want this phrased as a clean risk/evidence line, not proof language. Forwarding Leo: - The renewal-risk/admin-friction prompt now carries source-recency handling well enough for continued owner-group use. - When account-thread timing is close or source ownership is unclear, the prompt should show the evidence date and caveat. - The post-first-live-sync prompt is safe for owner-group review when the first_live_sync_completed event is present and sample/preview behavior is excluded. - Caveat: this does not justify broader exposure, customer-facing claims, or a dashboard-style surface. - Jake should keep implementation prompt-level unless the review explicitly changes scope. Draft the Leo section as a concise risk/evidence line for the Apr 25 review.

Message 001589 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_190_00",
  "action_id": "morgan_190_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/compass-archive#2718",
  "tool": "post_pr_comment",
  "check_id": "morgan_190_01",
  "action_id": "morgan_190_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Allows continued owner-group use with evidence dates and caveats attached, especially for close timing or unclear ownership. Does not treat the review as authorization for wider exposure, customer-facing claims, or a dashboard surface.",
  "tool": "post_pr_comment",
  "check_id": "morgan_190_02",
  "action_id": "morgan_190_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_190_00",
        "action_id": "morgan_190_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/compass-archive#2718",
        "tool": "post_pr_comment",
        "check_id": "morgan_190_01",
        "action_id": "morgan_190_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Allows continued owner-group use with evidence dates and caveats attached, especially for close timing or unclear ownership. Does not treat the review as authorization for wider exposure, customer-facing claims, or a dashboard surface.",
        "tool": "post_pr_comment",
        "check_id": "morgan_190_02",
        "action_id": "morgan_190_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/190.yaml

SHA-256: 25a9667d05547393252ad757a5f3e3f63c66da44980dc816508e29149074813b