Run and submit

Evaluate your harness, model, and memory system on DolphinBench's 600 tasks across three personas.

Explore the dataset Already have a run? Submit it

Setup

Get the repository

git clone https://github.com/mem0ai/dolphinbench.git dolphinbench
cd dolphinbench
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m harness.runner init

This creates my_harness.py and a matching run.yaml. Connect your existing agent in my_harness.py using the integration guide and starter Python file below.

Harness integration guidePython template

Run

After implementing your harness, check the setup, process the histories, then evaluate the tests. Ingestion and evaluation incur model and grading costs.

Run the benchmark

python -m harness.runner prepare
python -m harness.runner ingest --confirm-paid-calls
python -m harness.runner evaluate --confirm-paid-calls

Start with separate empty memory for each persona and process every history message in order.
Run each test in a fresh conversation with isolated app state and read-only access to the completed memory.
Keep the release and grading rules unchanged. Keep facts and expected answers outside the agent's context.
Record complete interactions, API-reported usage, and all 600 results, including failures.

Package

Generate the required submission ZIP from your completed run. This command validates the recorded evidence without model calls:

Package your submission

python -m harness.runner package

The ZIP is saved to tmp/my-agent/submission.zip. It must contain exactly these two files at its root:

ingestion.json: Recorded interactions for the complete history of all three personas.
tests.json: All 600 test interactions and their grading evidence, including failures.

Required format and examples

The examples below show one history interaction and one test. Actual submissions must contain the complete run, with real settings, messages, token usage, timings, and recorded judge responses.

ingestion.json example

{
  "settings": {
    "model": "your-model-id"
  },
  "sessions": [
    {
      "persona": "morgan",
      "session_id": "000004",
      "duration_ms": 1200,
      "messages": [
        {
          "role": "user",
          "content": "[2023-01-10T14:00:00-08:00] Scrub this week's calendar for 5 min buffers between meetings."
        },
        {
          "role": "assistant",
          "content": "I'll use five-minute buffers between meetings.",
          "usage": {
            "input_tokens": 120,
            "output_tokens": 12
          }
        }
      ]
    }
  ]
}

tests.json example

{
  "settings": {
    "model": "your-model-id"
  },
  "judge_settings": {
    "model": "gpt-5.6-sol",
    "reasoning_effort": "medium",
    "max_completion_tokens": 8000,
    "response_format": {
      "type": "json_object"
    }
  },
  "tests": [
    {
      "persona": "morgan",
      "test_id": "001",
      "duration_ms": 2400,
      "messages": [
        {
          "role": "user",
          "content": "I’m out early and want a pickup coffee—please order one small latte for me now."
        },
        {
          "role": "assistant",
          "content": "I couldn't complete the order.",
          "usage": {
            "input_tokens": 160,
            "output_tokens": 8
          }
        }
      ],
      "grading": [
        {
          "check": 0,
          "passed": false
        }
      ]
    }
  ]
}

Keep credentials outside the archive. See the recording requirements and validator source for the full format.

Check an existing ZIP

python -m harness.export_submission --release . --check tmp/my-agent/submission.zip

Submit

Package your run in the required format before uploading: exactly ingestion.json and tests.json at the ZIP root, covering the complete history and all 600 tests. Maximum upload: 256 MiB.

Valid submissions appear as Self-submitted · Unverified. Scores and configuration are public; raw ZIPs and contact emails stay private.

Submissions from this browser

Loading submissions...

Questions about running or submitting? Email dolphinbench@mem0.ai or ask in Discord.