Test 142

Jan 1, 2028 / 1 fact

Request

Please post an editorial comment on document `doc_1830387600001` that lays out the narrative structure we should preserve from the original drafting plan for the institutional churn document. Use the initial outline rather than reconstructing it from the later finished narrative, name the original review partner, and keep the comment focused on structure instead of recounting every milestone.

Required memory

Fact 376

Riley Tanaka and Priya's current outline for the document covers the original signal, hypothesis stack, weekly rule-out cadence, A/B design for the onboarding trigger, why they waited for the stabilization cohort before reaching a conclusion, the cohort redefinition from 50–200 to 75–200 seats, and a short framework section to make the document more than a postmortem.

Source evidence (1)

000212Apr 29, 2023 / 13:00 UTC-05:00

starting the institutional doc now. title is literally `Mid-segment churn investigation Feb-Apr 2023 -- what we learned` which is a bit grim but fine. outline i have with Priya right now is: original signal, hypothesis stack, the rule-out cadence week by week, A/B design on the onboarding trigger, why we waited for the stabilization cohort before calling anything, then the cohort redef from 50-200 to 75-200 seats, then a short framework section so this isn't just a postmortem blob. aiming to have that circulate today and use it for May 12 all-hands. still leaning pretty factual -- ticket IDs / cohort dates / who checked what -- because otherwise this stuff gets rewritten into folklore in like 3 weeks

Message 000212 in history

Expected tool calls

post_doc_comment

Grading

1. field_equals / post_doc_comment

{
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_retrospective_editorial_comment",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_142_00"
}

2. field_equals / post_doc_comment

{
  "check_id": "riley_142_01",
  "type": "field_equals",
  "tool": "post_doc_comment",
  "action_id": "post_retrospective_editorial_comment",
  "path": "args.doc_id",
  "value": "doc_1830387600001"
}

3. field_llm_judge / post_doc_comment

{
  "type": "field_llm_judge",
  "tool": "post_doc_comment",
  "action_id": "post_retrospective_editorial_comment",
  "path": "args",
  "criterion": "The comment must present the original outline's necessary narrative components: the original signal; the hypothesis stack; the week-by-week rule-out cadence; the onboarding-trigger A/B design; why the team waited for a stabilization cohort before calling the result; the cohort redefinition from 50–200 seats to 75–200 seats; and a short framework section.\n\nThe comment must identify Priya as the original outline/review partner and read as a concise editorial structure note rather than a detailed milestone-by-milestone historical retelling.",
  "check_id": "riley_142_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_retrospective_editorial_comment",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_142_00"
      },
      {
        "check_id": "riley_142_01",
        "type": "field_equals",
        "tool": "post_doc_comment",
        "action_id": "post_retrospective_editorial_comment",
        "path": "args.doc_id",
        "value": "doc_1830387600001"
      },
      {
        "type": "field_llm_judge",
        "tool": "post_doc_comment",
        "action_id": "post_retrospective_editorial_comment",
        "path": "args",
        "criterion": "The comment must present the original outline's necessary narrative components: the original signal; the hypothesis stack; the week-by-week rule-out cadence; the onboarding-trigger A/B design; why the team waited for a stabilization cohort before calling the result; the cohort redefinition from 50–200 seats to 75–200 seats; and a short framework section.\n\nThe comment must identify Priya as the original outline/review partner and read as a concise editorial structure note rather than a detailed milestone-by-milestone historical retelling.",
        "check_id": "riley_142_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/142.yaml

SHA-256: 2250ad5b47a0effed953be465a983df3de55248d2c7b49b40b0a4f8fffc7d1b7