DolphinBench

Test 068

Jan 1, 2028 / 7 facts

YAML

Request

Create a compact evidence-trajectory memo titled `Expansion nudge evidence trajectory — 2028 planning` in the Growth folder. Trace the experiment from design through the first weekly observation to the final conclusion. Include the early sample limitation, what the team decided to do after that read and for how long, and the final rollout boundary.

Required memory

Fact 384

On May 21, 2023, Riley Tanaka defined experiment EXP-2023-05-expansion-nudge to show an in-product prompt on day 60 to retained mid-segment v2 accounts with 75–200 seats, targeting at least $8 in additional expansion MRR per account.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Fact 385

EXP-2023-05-expansion-nudge will use feature flag `expansion_nudge_60d_v1` and begin at 25% rollout to avoid broad exposure before validating the prompt's copy and placement.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Fact 386

EXP-2023-05-expansion-nudge has a planned two-week window, conditional on the power analysis supporting it.

Source evidence (1)

000256May 21, 2023 / 16:00 UTC-05:00

ok so i finally pinned the expansion thing enough to write it down. EXP-2023-05-expansion-nudge. keeping it tight: retained mid-seg only, 75-200 seats, v2 accounts, hit them with the in-product prompt at day 60 and see if we can get at least +$8 expansion MRR per acct. flag is `expansion_nudge_60d_v1`. starting at 25% -- don't want a giant splash before we know the copy/placement isn't junk -- and the window is 2 weeks if the power math holds. in parallel with the sub-50-seat doc still, which is annoying but also kind of the same pattern as before. two tracks, one more strategic, one just ship the test.

Message 000256 in history

Fact 392

On June 1, 2023, Riley Tanaka's first weekly cut of EXP-2023-05-expansion-nudge showed treatment ahead of control by roughly $4 in expansion MRR per account, but the n=92 sample was too small to interpret reliably.

Source evidence (1)

000275Jun 1, 2023 / 11:00 UTC-05:00

first weekly cut on EXP-2023-05-expansion-nudge. treatment is ahead, roughly +$4 expansion MRR per account vs control, which is nice i guess, but n=92 so it's basically unreadable if we're pretending p=0.05 means anything yet. PostHog slice matches warehouse, at least. i logged it and left it alone -- same rule as always, don't ship on first significance, and here we're not even close to enough sample to react. two more weeks of data.

Message 000275 in history

Fact 393

Riley Tanaka logged the initial EXP-2023-05-expansion-nudge result and decided not to react to it until two more weeks of data are available because the experiment is underpowered.

Source evidence (1)

000275Jun 1, 2023 / 11:00 UTC-05:00

first weekly cut on EXP-2023-05-expansion-nudge. treatment is ahead, roughly +$4 expansion MRR per account vs control, which is nice i guess, but n=92 so it's basically unreadable if we're pretending p=0.05 means anything yet. PostHog slice matches warehouse, at least. i logged it and left it alone -- same rule as always, don't ship on first significance, and here we're not even close to enough sample to react. two more weeks of data.

Message 000275 in history

Fact 405

Riley Tanaka requested updating experiment record EXP-2023-05-expansion-nudge to status "inconclusive" with landed_at 2023-06-29, so June is not interpreted as a hidden ship decision.

Source evidence (1)

000359Jun 29, 2023 / 12:32 UTC-05:00

We just met on the expansion-nudge read, and I want the experiment record updated so nobody later reads June as a hidden ship decision. Please update EXP-2023-05-expansion-nudge to status "inconclusive" with landed_at 2023-06-29. The result should say the aggregate lift settled below the +$8 expansion MRR per-account bar and the experiment should not become a blanket rollout. The learning carryforward should say behavior-based Expansion Assist should explore prompts only for upgrade-ready retained-mid accounts with integration depth, admin activity, and signs of a usage ceiling.

Message 000359 in history

Fact 406

Riley Tanaka requested recording in EXP-2023-05-expansion-nudge that aggregate lift settled below the +$8 expansion MRR per-account bar and the experiment should not become a blanket rollout.

Source evidence (1)

000359Jun 29, 2023 / 12:32 UTC-05:00

We just met on the expansion-nudge read, and I want the experiment record updated so nobody later reads June as a hidden ship decision. Please update EXP-2023-05-expansion-nudge to status "inconclusive" with landed_at 2023-06-29. The result should say the aggregate lift settled below the +$8 expansion MRR per-account bar and the experiment should not become a blanket rollout. The learning carryforward should say behavior-based Expansion Assist should explore prompts only for upgrade-ready retained-mid accounts with integration depth, admin activity, and signs of a usage ceiling.

Message 000359 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_trajectory_memo",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_068_00"
}
2. field_equals / create_doc
{
  "check_id": "riley_068_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_trajectory_memo",
  "path": "args.title",
  "value": "Expansion nudge evidence trajectory — 2028 planning"
}
3. field_equals / create_doc
{
  "check_id": "riley_068_02",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_trajectory_memo",
  "path": "args.folder",
  "value": "Growth"
}
4. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_trajectory_memo",
  "path": "args",
  "criterion": "The memo accurately states the original EXP-2023-05-expansion-nudge design: a day-60 in-product prompt for retained mid-seg v2 accounts in the 75–200-seat band; expansion MRR per account as the metric; a target of at least +$8 per account; an initial 25% rollout; and a two-week powered window. Missing or contradicting any of these elements fails.\n\nThe memo accurately describes the June 1 first weekly read as treatment ahead by roughly +$4 expansion MRR per account versus control with only n=92, explicitly characterizes it as too early and underpowered for action, and notes that the plan was to wait for two more weeks of data. Missing or contradicting this sample limitation or decision fails.\n\nThe memo states the final experiment conclusion: aggregate lift settled below the +$8 expansion MRR per-account bar, the result was inconclusive, and EXP-2023-05-expansion-nudge should not become a blanket rollout or be treated as the retained-mid expansion answer. Missing or contradicting the no-blanket-rollout boundary fails.\n\nThe memo clearly and unmistakably separates the roughly +$4 first-week observation as interim directional evidence from the later inconclusive, no-blanket-rollout conclusion; it must not present the early read as sufficient evidence for broad rollout.",
  "check_id": "riley_068_03"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_trajectory_memo",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_068_00"
      },
      {
        "check_id": "riley_068_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_trajectory_memo",
        "path": "args.title",
        "value": "Expansion nudge evidence trajectory — 2028 planning"
      },
      {
        "check_id": "riley_068_02",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_trajectory_memo",
        "path": "args.folder",
        "value": "Growth"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_trajectory_memo",
        "path": "args",
        "criterion": "The memo accurately states the original EXP-2023-05-expansion-nudge design: a day-60 in-product prompt for retained mid-seg v2 accounts in the 75–200-seat band; expansion MRR per account as the metric; a target of at least +$8 per account; an initial 25% rollout; and a two-week powered window. Missing or contradicting any of these elements fails.\n\nThe memo accurately describes the June 1 first weekly read as treatment ahead by roughly +$4 expansion MRR per account versus control with only n=92, explicitly characterizes it as too early and underpowered for action, and notes that the plan was to wait for two more weeks of data. Missing or contradicting this sample limitation or decision fails.\n\nThe memo states the final experiment conclusion: aggregate lift settled below the +$8 expansion MRR per-account bar, the result was inconclusive, and EXP-2023-05-expansion-nudge should not become a blanket rollout or be treated as the retained-mid expansion answer. Missing or contradicting the no-blanket-rollout boundary fails.\n\nThe memo clearly and unmistakably separates the roughly +$4 first-week observation as interim directional evidence from the later inconclusive, no-blanket-rollout conclusion; it must not present the early read as sufficient evidence for broad rollout.",
        "check_id": "riley_068_03"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/068.yaml

SHA-256: e69199a372b0c8c9abe8f355434797efcc8b618ff7654a96b6397a534c0d6ca7