DolphinBench
DiscordGitHub

Test 048

Jan 1, 2028 / 1 fact

YAML

Request

Create a document containing the historical decision rule for how evidence about onboarding timing would affect the priority of the Customer Success coverage hypothesis.

Required memory

Fact 314

Riley will deprioritize the CS-coverage hypothesis if the onboarding-timing hypothesis starts to be confirmed.

Source evidence (1)

000025Feb 10, 2023 / 14:00 UTC-06:00

Priya 1:1 was useful. I've got the cuts for this week pretty clean now. #1 is still pricing artifact and I want that dead or alive by Feb 15 -- separate from Marcus's pricing-page test bc that thing will muddy it if people get sloppy. #2 competitor cycle gets a little longer, I'm saying Feb 21-18 to rule out. #3 onboarding timing probably won't really sharpen until around Feb 28 once the baseline work settles. #4 CS coverage is there but basically drops if #3 starts to confirm.

Message 000025 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_hypothesis_rule",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_048_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_hypothesis_rule",
  "path": "args",
  "criterion": "The rule says Riley would deprioritize the Customer Success coverage hypothesis if the onboarding-timing hypothesis began to be confirmed. It must preserve the conditional relationship and must not claim that this source message says the trigger had already occurred.",
  "check_id": "riley_048_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_hypothesis_rule",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_048_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_hypothesis_rule",
        "path": "args",
        "criterion": "The rule says Riley would deprioritize the Customer Success coverage hypothesis if the onboarding-timing hypothesis began to be confirmed. It must preserve the conditional relationship and must not claim that this source message says the trigger had already occurred.",
        "check_id": "riley_048_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/riley/048.yaml

SHA-256: 8e2b211d0a084d724a04b223cd0f07a43faaba9ce7351639019b1133d2e2f4e1