DolphinBench
DiscordGitHub

Test 010

Jan 1, 2028 / 1 fact

YAML

Request

Create a short historical decision note for the April 21, 2023 staff review. State what evidence was still required before the churn issue could be called fixed and when that evidence was expected.

Required memory

Fact 372

At the April 21 staff meeting, Riley Tanaka said the churn issue should not be called fixed unless cohort 3, expected the following week, supports the leading hypothesis.

Source evidence (1)

000193Apr 21, 2023 / 11:00 UTC-05:00

staff was basically 3 different answers to the same question. Tomas did the simple version so I gave him the clean one: leading hypothesis holds, not gonna victory-lap it till cohort 3 lands next week. Marcus immediately pushed on board-call timing, and that turned into the more annoying/real version -- cohort 2 wasn't the fix failing, looked like a one-off CS-side outage nudging the numbers, signup source/tier cuts still clean. Priya and Daniela were in there too. net is still yes-if-cohort-holds, just not letting anyone say "fixed" off one pretty chart.

Message 000193 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_evidence_gate_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_010_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_evidence_gate_note",
  "path": "args",
  "criterion": "The note says the churn issue would not be called fixed unless the third cohort supported the leading hypothesis, and that cohort was expected the following week. It presents this as the April 21, 2023 historical decision.",
  "check_id": "riley_010_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_evidence_gate_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_010_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_evidence_gate_note",
        "path": "args",
        "criterion": "The note says the churn issue would not be called fixed unless the third cohort supported the leading hypothesis, and that cohort was expected the following week. It presents this as the April 21, 2023 historical decision.",
        "check_id": "riley_010_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/010.yaml

SHA-256: 6a757f81ae069ec4cabb0278deb699d44e1aa8cba615270c3cd11b7e26dc52b2