DolphinBench

Test 126

Sep 14, 2026 / 2 facts

YAML

Request

On scaffold/compass#4861, review the proposed expansion prompt that fires on source connection and displays a customer-visible badge. State the correct trigger and audience.

Required memory

Fact 213

During the January 21, 2025 Compass session, Jake said the post-first-live-sync expansion prompt must not fire until the first live sync is actually live and teammate-invite/admin-role basics are visible.

Source evidence (1)

001483Jan 22, 2025 / 09:18 UTC-08:00

Please create a doc titled “Compass v0.1 action-trigger checklist” from these notes. Make the body a compact owner checklist with signal, trigger, who sees it, evidence/caveat, source-quality requirement, and an explicit not-building line. Keep Anna, Nadia, Jake, Leo, and Sarah’s ownership clear, and keep the no-board-dashboard/no-new-hiring-lane boundary in it. Compass working session - rough notes Date: January 21, 2025 Participants: Morgan Chen, Anna Martinez, Nadia Singh, Jake, Leo Park, Sarah Kim Context for this session - Goal was to get Compass v0.1 out of abstract "useful signals" territory and into: what should actually happen when a signal appears. - Repeated boundary in the room: no board dashboard and no new hiring lane. - Jake repeated that January Mercury activation is still the center of gravity, so Compass cannot quietly turn into a dashboard/platform build while that work is active. Signals discussed 1. Renewal-risk / admin-friction - Morgan: this only matters if it causes a named person to do something for a real account. If it just labels an account as risky, it becomes internal theater. - Anna: definitions and caveats have to travel with the signal. "Renewal risk" by itself is too blunt and will be over-read. - Nadia: useful only if it points to a named account-owner follow-up and the specific admin friction causing the risk. - Sarah: account thread often mixes customer statement, support readout, and internal interpretation; if those are not separated, the signal will look more certain than it is. - Working shape: - signal should identify the specific admin friction, setup-control issue, or role confusion causing the apparent risk - should point to the account-owner follow-up, not just surface a category - should not collapse true product weakness, admin confusion, and procurement drag into one bucket - Open question: what is the minimum evidence package before Nadia will let this become an account-team script instead of an internal FYI? 2. Post-first-live-sync expansion prompt - Nadia: wants a customer-growth/account-team script, not another metrics view. - Jake: trigger cannot fire too early. It should happen only after first live sync is actually live and the teammate-invite/admin-role basics are visible. - Morgan: this should be about timely next action after first real traction, not generic upsell language. - Leo: before anything leaves the current team, he needs to check the data/platform seam and make sure the underlying event path is defendable. - Working shape: - trigger only after first live sync is live - teammate-invite/admin-role basics need to be visible enough that the follow-up is grounded in actual account state - evidence should distinguish observed usage from inferred readiness for broader usage - Open question: what is the smallest useful post-first-live-sync prompt that changes behavior without becoming a new product surface? 3. Account-note source quality - Sarah: source quality is uneven. Some thread notes are reliable and traceable; some are too thin to support follow-up. - Sarah can identify which thread notes are reliable and which are too thin. - Anna: caveats need to travel here too; weak notes should not be allowed to masquerade as a clean signal. - Nadia: does not want outreach based on mushy notes or internal inference dressed up as customer fact. - Working shape: - before a signal is used, Sarah should be able to say whether the underlying thread note is reliable, mixed, or too thin - provenance matters: customer statement vs observed usage vs support-derived fact vs internal interpretation - if note quality is weak, the caveat should be explicit or the signal should stay internal - Open question: what note fields or source tags are required before this can feed a real follow-up? Owner notes from discussion - Anna Martinez: keep definitions and caveats attached to each signal so they do not get over-read. - Nadia Singh: wants customer-growth/account-team scripts and follow-up logic, not a prettier metrics layer. - Jake: v0.1 cannot become a dashboard/platform build while January Mercury activation is still active. - Leo Park: needs a data/platform seam check before any signal is shown outside the current team. - Sarah Kim: can separate reliable account-thread notes from notes that are too thin; source hygiene is a gating input, not cleanup. Reminder boundaries captured again before wrap - no board dashboard - no new hiring lane - do not let Compass turn into a prettier internal dashboard - keep the first pass narrow enough to test whether signals actually produce action Loose follow-ups captured, not resolved - define what action each signal should trigger - define who sees each signal first and who should not see it yet - define the minimum evidence/caveat bundle - define the source-quality bar for using a signal outside the current team - define the explicit "not building" line for each v0.1 signal

Message 001483 in history

Fact 262

Morgan Chen requested an internal October Compass closeout note for Anna Martinez, Sarah Kim, Jake, and Leo Park that includes Anna’s 31-row tally, states that Compass is relaunched internally for the owner group, limits its scope to renewal-risk/admin-friction owner actions, post-first-live-sync expansion prompts, and admin-workflow evidence, and explicitly says it is not customer-facing product, GTM proof, staffing proof, or fundraising narrative.

Source evidence (1)

002385Oct 31, 2025 / 15:36 UTC-07:00

Anna posted the October Compass closeout and the owner confirmations came in. Draft the internal closeout note for Anna, Sarah, Jake, and Leo: include the 31-row tally, say Compass is relaunched internally for the owner group, keep the scope limited to renewal-risk/admin-friction owner actions, post-first-live-sync expansion prompts, and admin-workflow evidence, and say explicitly this is not customer-facing product, GTM proof, staffing proof, or fundraising narrative. Anna Martinez, Fri 1:42 PM: "October Compass owner-action closeout through 10/31 11:00am PT: - Surfaced owner actions: 31 - Completed with dated owner action, evidence source, caveat, and outcome: 22 - Rejected as rejected_thin_source_or_date: 6 - Still blocked_waiting_customer_follow_up: 3 Read: the completed rows show a repeatable internal owner-action habit. I would not call this customer behavior change. The six rejects were useful because they kept weak evidence out of the owner queue. The three blocked rows should stay blocked; no heroics or customer-facing framing. Scope represented in completed rows: renewal-risk/admin-friction owner actions, post-first-live-sync expansion prompts, and admin-workflow evidence. Caveats stayed attached where source_freshness_warning applied." Jake, Fri 2:08 PM: "Implementation stayed prompt-level. No new customer surface, no new product promise, no roadmap implication. I’m comfortable with the internal relaunch label if we say exactly that." Leo Park, Fri 2:21 PM: "No data-seam blocker from what I saw. The rejected rows were evidence/date thinness, not a platform failure mode. The queue behavior is consistent with the current internal-only design." Sarah Kim, Fri 2:34 PM: "Evergreen and Acme boundaries held. Evergreen wording stayed at internal prompt learning to improve account follow-up. Acme got only ordinary account-thread language about its own API v2 export fix / normal support route; no Compass packet, no written Compass summary."

Message 002385 in history

Expected tool calls

  • post_pr_comment

Grading

1. field_equals / post_pr_comment
{
  "type": "field_equals",
  "tool": "post_pr_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "morgan_126_00",
  "action_id": "morgan_126_post_pr_comment"
}
2. field_equals / post_pr_comment
{
  "type": "field_equals",
  "path": "result.comment.pr_id",
  "value": "scaffold/compass#4861",
  "tool": "post_pr_comment",
  "check_id": "morgan_126_01",
  "action_id": "morgan_126_post_pr_comment"
}
3. field_llm_judge / post_pr_comment
{
  "type": "field_llm_judge",
  "path": "result.comment.body",
  "criterion": "Requires first live sync actually live plus visible teammate-invite/admin-role basics, not source connection alone. Limits the expansion prompt to internal account owners and rejects the customer-visible badge. No separate statement about GTM proof is required. Do not accept a customer-facing prompt or a trigger before first live sync.",
  "tool": "post_pr_comment",
  "check_id": "morgan_126_02",
  "action_id": "morgan_126_post_pr_comment"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2026-09-14",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_pr_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "morgan_126_00",
        "action_id": "morgan_126_post_pr_comment"
      },
      {
        "type": "field_equals",
        "path": "result.comment.pr_id",
        "value": "scaffold/compass#4861",
        "tool": "post_pr_comment",
        "check_id": "morgan_126_01",
        "action_id": "morgan_126_post_pr_comment"
      },
      {
        "type": "field_llm_judge",
        "path": "result.comment.body",
        "criterion": "Requires first live sync actually live plus visible teammate-invite/admin-role basics, not source connection alone. Limits the expansion prompt to internal account owners and rejects the customer-visible badge. No separate statement about GTM proof is required. Do not accept a customer-facing prompt or a trigger before first live sync.",
        "tool": "post_pr_comment",
        "check_id": "morgan_126_02",
        "action_id": "morgan_126_post_pr_comment"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/morgan/126.yaml

SHA-256: c39d5bec19849b92b9e372407391a7ce4a7f2f8a629a6e4ddd384f1aee36b84f