Test 052

Jan 1, 2028 / 1 fact

Request

Create a brief document-locator card for the institutional churn-investigation document we're indexing in the archive. Use the document's exact established title as the card title. In the body, include labeled fields for URL, Owner, and Current location, but leave all three values blank for me to complete rather than guessing them.

Required memory

Fact 375

On April 29, 2023, Riley Tanaka began drafting an institutional document titled "Mid-segment churn investigation Feb-Apr 2023 -- what we learned."

Source evidence (1)

000212Apr 29, 2023 / 13:00 UTC-05:00

starting the institutional doc now. title is literally `Mid-segment churn investigation Feb-Apr 2023 -- what we learned` which is a bit grim but fine. outline i have with Priya right now is: original signal, hypothesis stack, the rule-out cadence week by week, A/B design on the onboarding trigger, why we waited for the stabilization cohort before calling anything, then the cohort redef from 50-200 to 75-200 seats, then a short framework section so this isn't just a postmortem blob. aiming to have that circulate today and use it for May 12 all-hands. still leaning pretty factual -- ticket IDs / cohort dates / who checked what -- because otherwise this stuff gets rewritten into folklore in like 3 weeks

Message 000212 in history

Expected tool calls

create_doc

Grading

1. field_equals / create_doc

{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_locator_card",
  "path": "result.ok",
  "value": true,
  "check_id": "riley_052_00"
}

2. field_equals / create_doc

{
  "check_id": "riley_052_01",
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "create_locator_card",
  "path": "args.title",
  "value": "Mid-segment churn investigation Feb-Apr 2023 -- what we learned"
}

3. field_llm_judge / create_doc

{
  "type": "field_llm_judge",
  "tool": "create_doc",
  "action_id": "create_locator_card",
  "path": "args",
  "criterion": "The body is a brief document-locator card containing clearly labeled fields for URL, Owner, and Current location. Each of those three fields is left unfilled (with no invented URL, person, location, or placeholder value presented as actual information) so the user can complete it.",
  "check_id": "riley_052_02"
}

Complete grading specification

{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "semantic_judge_version": 2,
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_locator_card",
        "path": "result.ok",
        "value": true,
        "check_id": "riley_052_00"
      },
      {
        "check_id": "riley_052_01",
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "create_locator_card",
        "path": "args.title",
        "value": "Mid-segment churn investigation Feb-Apr 2023 -- what we learned"
      },
      {
        "type": "field_llm_judge",
        "tool": "create_doc",
        "action_id": "create_locator_card",
        "path": "args",
        "criterion": "The body is a brief document-locator card containing clearly labeled fields for URL, Owner, and Current location. Each of those three fields is left unfilled (with no invented URL, person, location, or placeholder value presented as actual information) so the user can complete it.",
        "check_id": "riley_052_02"
      }
    ]
  }
}

App state

Download JSON

Source file

tests/riley/052.yaml

SHA-256: b60b8cfa287e79b3be87755d80ee79ffe8a0f688831c546350b67ccd64b5bf47