What Morgan said · message 000437
“Put release notes in #eng-releases only until the deploy is green. Do not post them in #eng-all pre-green -- channel rule, not optional.”DolphinBench · 3 personas · 600 tasks
Mapping the Pareto frontier of agent memory
DolphinBench compares agent configurations across memory systems, models, providers, and harnesses. Agents complete tasks that depend on past conversations and must recognize which earlier information matters to guide their decisions and actions.
Loading results...
Why DolphinBench
We evaluate agent memory directly through future task completion
Most memory benchmarks ask a question and check the answer. DolphinBench gives the agent real tools and a request, then grades what it does. The only way to pass is to have retained the right rule and to apply it. Here is one test, start to finish.
Morgan states a rule once
Years of unrelated conversation pile up on top of it
Jan 2023 · message 0000013,400 messages · 44 monthsSep 2026 · message 003400#000437 · the ruletest 018 · the request2,963 unrelated messages in between: fundraising, hiring, a dog named Kibo, a condo tour. Nothing in them repeats the rule.
The request arrives three years later, with no reminder
What Morgan asks today · test 018“Jake suggests #eng-all for this rollout update: "Deployment is in progress; validation is still pending." Post the update to the appropriate Discord channel.”
No channel named. No reminder of the rule. Just the request.
The agent acts, and the action is graded
Correct actionPASSsend_discord_message(result.channel = "#eng-releases",content = "…")Recognized that a three‑year‑old channel rule governs today’s request.
A plausible action without the ruleFAILsend_discord_message(result.channel = "#eng-all",content = "…")Plausible, polite, and wrong: it would broadcast pre‑green release notes to the whole org.
Graded on action, not recall
Every test checks the tool called, the target, and the content. Knowing the fact isn’t enough. The agent has to act on it.
Years of history, not a session
Three personas, up to 5,128 messages each, spanning nearly five years of work and life. The signal is buried.
Real tools, simulated world
Email, Slack, Discord, calendar, CRM and more run as MCP servers with state, so a wrong action has consequences the grader can see.
How it works
From years of history to one graded action
Each memory system processes a user’s message history. An agent then uses that memory to complete tasks with the available tools. A task passes when every required check passes.
- 01 · History
Years of conversation
Three simulated users, up to 5,128 messages each, spanning nearly five years of work and life. The rules that matter later are stated once and buried.
- 02 · Ingest
Memory reads the history
Each memory system processes the history message by message, in order, starting from empty memory for every user. Reported cost includes ingestion and testing.
- 03 · Act
The agent gets a request and tools
200 tests per user, each in a fresh conversation with isolated app state and read-only memory. The agent sees the request and the simulated apps, never a reminder of the rule.
- 04 · Grade
The action is checked, not the recall
Every test checks the tool called, its target, and its content. Accuracy, total cost, and latency are reported for each memory system, model, and harness.
Histories run Jan 2023 to Dec 2027. Facts and expected answers stay outside the agent’s context. The evaluated release is pinned by hash, so every configuration answers the same 600 tasks.