DolphinBench

Test 051

Jan 1, 2028 / 2 facts

YAML

Request

I'm cleaning up the Q2 planning archive. Create a concise activation-target note dated April 5, 2023 that clearly separates the target that remained active for Q2 from the higher small-business-tier planning target and the condition that would have triggered it. Make it explicit which number should remain on the Q2 deck, and do not present the future scenario as an already-approved current target.

Required memory

Fact 356

Priya decided that the 60% activation target will continue into Q2, so Riley Tanaka is not revising the deck number yet.

Source evidence (1)

000152Apr 5, 2023 / 08:30 UTC-05:00

this morning is Ines + one more pass on the email template QA before the Apr 10 ship. want the boring stuff clean -- token fallbacks, locale weirdness, unsubscribe/footer on the old customer.io branches. also Priya kept the Q1 activation target at 60% into Q2 as-is in our 1:1, so i'm not re-cutting that deck number yet. if the sub-50 seat tier really ships in Q3 then we'll move it to 65% in the small-business-tier planning.

Message 000152 in history

Fact 357

Riley Tanaka plans to raise the small-business-tier activation target to 65% if the sub-50-seat tier ships in Q3.

Source evidence (1)

000152Apr 5, 2023 / 08:30 UTC-05:00

this morning is Ines + one more pass on the email template QA before the Apr 10 ship. want the boring stuff clean -- token fallbacks, locale weirdness, unsubscribe/footer on the old customer.io branches. also Priya kept the Q1 activation target at 60% into Q2 as-is in our 1:1, so i'm not re-cutting that deck number yet. if the sub-50 seat tier really ships in Q3 then we'll move it to 65% in the small-business-tier planning.

Message 000152 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_activation_target_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_051_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_activation_target_note",
  "path": "args",
  "criterion": "Across the title or body, the note is clearly dated April 5, 2023.\n\nThe document clearly states that the Q2 activation target remained unchanged at 60% and that 60% is the number that should remain on the Q2 deck.\n\nThe document clearly states that 65% was only a conditional small-business-tier planning target, applicable if the sub-50-seat tier actually shipped in Q3, and distinguishes it from the current Q2 target rather than presenting it as already approved or active.",
  "check_id": "riley_051_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_activation_target_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_051_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_activation_target_note",
        "path": "args",
        "criterion": "Across the title or body, the note is clearly dated April 5, 2023.\n\nThe document clearly states that the Q2 activation target remained unchanged at 60% and that 60% is the number that should remain on the Q2 deck.\n\nThe document clearly states that 65% was only a conditional small-business-tier planning target, applicable if the sub-50-seat tier actually shipped in Q3, and distinguishes it from the current Q2 target rather than presenting it as already approved or active.",
        "check_id": "riley_051_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/051.yaml

SHA-256: 4c667896d7f1302b5d828fd87c8b043593e0bc632b9d2e35e9694a725663ebbc