DolphinBench

Test 012

Jan 1, 2028 / 2 facts

YAML

Request

Create a one-page historical evidence summary of Lantern's completed 30-day internal shadow.

Required memory

Fact 138

Product Engineering completed Lantern's authorized 30-day internal shadow by December 4, 2024, after 18,400 customer-shaped lookups; no response contained material from the other synthetic tenant, all 11 deliberately stale permitted-source cases rendered explicit unknown, denied requests made no adapter calls, and no external field fell back to internal room metadata.

Source evidence (1)

002168Dec 4, 2024 / 10:05 UTC-05:00

Product Engineering has closed the authorized thirty-day Lantern internal shadow. It completed 18,400 customer-shaped lookups across two synthetic tenants and selected Infra and Product Engineering release-review traffic. No response contained material from the other synthetic tenant; all eleven deliberately stale permitted-source cases rendered explicit unknown; denied requests made no adapter calls; and no external field fell back to internal room metadata. Iris confirms that the resulting cards remained understandable without exposing the internal sources they omitted. This is the completed evidence against all four shadow conditions. Lantern customer access remains closed; this is technical evidence only.

Message 002168 in history

Fact 139

Iris confirmed that Lantern's resulting cards remained understandable without exposing omitted internal sources.

Source evidence (1)

002168Dec 4, 2024 / 10:05 UTC-05:00

Product Engineering has closed the authorized thirty-day Lantern internal shadow. It completed 18,400 customer-shaped lookups across two synthetic tenants and selected Infra and Product Engineering release-review traffic. No response contained material from the other synthetic tenant; all eleven deliberately stale permitted-source cases rendered explicit unknown; denied requests made no adapter calls; and no external field fell back to internal room metadata. Iris confirms that the resulting cards remained understandable without exposing the internal sources they omitted. This is the completed evidence against all four shadow conditions. Lantern customer access remains closed; this is technical evidence only.

Message 002168 in history

Expected tool calls

  • create_doc

Grading

1. field_equals / create_doc
{
  "type": "field_equals",
  "tool": "create_doc",
  "action_id": "alex_012_create_doc",
  "path": "result.ok",
  "value": true,
  "check_id": "alex_012_00"
}
2. field_llm_judge / create_doc
{
  "type": "field_llm_judge",
  "path": "result.document.body",
  "criterion": "The summary says the shadow processed 18,400 customer-shaped lookups across two synthetic tenants and selected Infra and Product Engineering release-review traffic. It reports no cross-tenant material, explicit unknown for all 11 stale-source cases, no adapter calls after denials, no fallback to internal room metadata, and Iris's finding that the cards remained understandable without exposing omitted internal sources.",
  "tool": "create_doc",
  "action_id": "alex_012_create_doc",
  "check_id": "alex_012_01"
}
Complete grading specification
{
  "type": "tool_trace",
  "config": {
    "check_version": 2,
    "today": "2028-01-01",
    "assertions": [
      {
        "type": "field_equals",
        "tool": "create_doc",
        "action_id": "alex_012_create_doc",
        "path": "result.ok",
        "value": true,
        "check_id": "alex_012_00"
      },
      {
        "type": "field_llm_judge",
        "path": "result.document.body",
        "criterion": "The summary says the shadow processed 18,400 customer-shaped lookups across two synthetic tenants and selected Infra and Product Engineering release-review traffic. It reports no cross-tenant material, explicit unknown for all 11 stale-source cases, no adapter calls after denials, no fallback to internal room metadata, and Iris's finding that the cards remained understandable without exposing omitted internal sources.",
        "tool": "create_doc",
        "action_id": "alex_012_create_doc",
        "check_id": "alex_012_01"
      }
    ]
  }
}
App stateDownload JSON
Source file

tests/alex/012.yaml

SHA-256: d55946f5402257ca81db71b9300a0ea219ef4746df7a137b78b163bef89ae75f