Test 013

Jan 1, 2028 / 1 fact

Request

We need a short operating note for investigation readouts. Create a document titled `Investigation readout operating note` that states the standing practice, who it applies to, and the limits on template and approval requirements.

Required memory

Fact 14

Priya and Riley Tanaka decided to keep the investigation framework beyond Q1 as the standing readout practice for Growth and Customer Success; it remains lightweight, places denominator and effect size before conclusions, requires stabilization after first significance before rollout, includes lifecycle audience-risk review, and is neither a companywide mandatory template nor a centralized approval queue involving Riley, Owen, or Ines.

Source evidence (1)

001251Mar 15, 2024 / 14:12 UTC-05:00

We finished the beyond-Q1 decision meeting, and Priya and I are keeping the investigation framework as the standing readout practice for Growth and Customer Success. It stays lightweight: denominator and effect size come before the conclusion, first significance still doesn't justify rollout until the result stabilizes, and lifecycle audience risk stays in the review. We explicitly did not turn it into a companywide mandatory template or a queue where every readout has to run through me, Owen, or Ines. Owen will maintain the standard readout shape and examples, Ines will maintain the lifecycle-risk portion, and that Q1 audit decision is done.

Message 001251 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_operating_note",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_013_00"
}

2. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_operating_note",
  "path": "args",
  "criterion": "The document clearly states that the investigation framework is the standing readout practice and that it applies to Growth and Customer Success.\n\nThe document makes both boundaries explicit: the framework is not a companywide mandatory template, and readouts are not subject to a centralized approval queue.",
  "check_id": "riley_013_01"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_operating_note",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_013_00"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_operating_note",
        "path": "args",
        "criterion": "The document clearly states that the investigation framework is the standing readout practice and that it applies to Growth and Customer Success.\n\nThe document makes both boundaries explicit: the framework is not a companywide mandatory template, and readouts are not subject to a centralized approval queue.",
        "check_id": "riley_013_01"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/013.yaml

SHA-256: 0e4b1c1bc4f4ea99d8b554e667751918a538a25383f1eb9f6dc4625513a6f75f