We beat Mem0 on their own benchmark, on their own terms. Now we can't submit it.

Mem0 published DolphinBench in September. It is a benchmark for agent memory: three fictional people, 13,539 messages of their history, and 600 tests where an agent has to remember the right thing and then actually do something about it in a simulated app. Most memory benchmarks ask "did you recall the fact". This one asks "did you post the comment, book the slot, send the one message the person wanted". I liked that. It is the closest thing to how people actually use Empirical.

So we ran it. All of it, the official way. It took six weeks, thirteen attempts, one cloud account that switched itself off mid-run, one desktop PC that ran out of memory, one old server dragged out of a closet, and a submission format that turned out not to fit a real run. We finished with 451 out of 600, which is ahead of every row on the published leaderboard that used the same agent.

This is the story, with receipts. It is also a request, because the number is not on the board yet, and the reason it is not on the board is the most interesting part.

The result

The leaderboard rows that count are the ones run with Hermes, the reference agent, driving GPT 5.6 Luna. That is the setup Mem0 used for their own row and for every other published system. Here is where we landed against them.

Each persona is 200 tests. We lead on Morgan and Alex outright. On Riley we are two tests behind Hermes's own built-in memory, which is a fair thing to be beaten by, since the built-in path has no network hop and no second system to disagree with. Cost is in the same band as Mem0's row. Latency is middle of the pack.

Two more rows exist that are not in that table: Graphon (504) and AtMem (479). Both were self-submitted, like ours, and both ran their own lightweight harness instead of Hermes. That is allowed, and it is a smart move, which I will come back to. But a number produced by a different agent is a different experiment, so I keep them out of the Hermes comparison.

Origin story

Ok. Let's go back to how this started, how we got here, and something you can do to help us out.

A friend of mine sent me the benchmark. I got interested to see how Empirical would do. That turned out to be a glorp fest and a massive rabbit hole.

I started the runs locally, on my desktop, which runs Windows. The runner died on the first file lock, because it uses a Unix locking call Windows does not have, and the evening went to getting it to start inside a Docker bind-mount. My friend hit the exact same wall the same evening, which was comforting in the way that misery is.

I created a shared workspace for Morgan, one of the three fictional people, so her whole history would live in one place I could look at. This is her brain map today, every memory the benchmark's history produced, grouped by what it is about:

Workspaces are members-only, so there is no public link, but I can share it with you if you want to poke around. Even getting it to work was a headache, because the benchmark wants one memory store per persona and a personal workspace cannot be shared. Then came the fighting with Hermes, the benchmark's reference agent. After hours of failures and wasted tokens I vibe coded my own adapter. I knew it was against the submission rules, which want the official runner driving the official agent, but at that point I just wanted something to work. And it did. All the memories went in, and I could test.

Those first tests were very promising. Hard to draw a clean line against Mem0's numbers, because my harness was not their harness, but Empirical was scoring well, and that was enough to make me want to keep going.

Head in the cloud

The next part was going to be expensive. To be comparable to the published rows it has to be run end to end, with no interruptions, on the exact setup the benchmark specifies: Hermes driving GPT 5.6 Luna, on Azure. The benchmark also starts a fresh agent container for every single message and needs the Docker socket to do it, so none of the managed container services will take it. It needs a real Linux box. Another rabbit hole.

So it is now the beginning of October. The VM is set up. The Azure account is funded. A two-message dry run passes three tests. Ready to start ingestion.

Failure. And failure. And failure.

Ever tried. Ever failed. No matter. Try again. Fail again. Fail better. Samuel Beckett, who never had to run a benchmark.

The rules are strict in a way I came to respect. Every message the model saw has to be recorded exactly as it saw it, with the API's own token counts. No rewriting, no tidying. And you may not change Hermes in any way that alters what the model sees. Fair enough, that is what makes the rows comparable.

But Hermes has many, many holes. That is not fair to say about Hermes directly; it is fair to say about the version of Hermes this benchmark pins. Hermes is built for a person at a terminal, not for a lab that records everything, and it has habits that make sense for the first and are poison for the second. When the model comes back with nothing, Hermes quietly asks again and pretends the first time never happened. When it wants to steer the model, it slips in a message dressed up as if the user typed it. When a long turn ends, it starts a second review in the background that the benchmark cuts off at the knees. Every now and then it drops an entire response on the floor because of a bookkeeping error inside its own recorder. Each of these is harmless in a terminal. Each one makes the transcript a lie, because it shows the model something it never saw, or hides something it did.

And the rules say you cannot just fix Hermes. So the work was the opposite of fixing: catch every one of these as it happens, write it down exactly as it happened, label it, and explain it in the method note, so a reader can see the seam instead of being fooled by it. Pin the exact Hermes version so nothing moves under you. Teach it to stop downloading its own dependencies fresh for every one of thirteen thousand messages. Price the Azure calls yourself, because that Hermes cannot. None of this is clever. It is the kind of work where every time you think you are done, the run gets a little further and finds the next one.

October 6th: the first full run started at 2:39 in the morning and stopped at 135 messages. Run two at 2:55. Run three at 4:22. Run four at 11:17, after Azure's content filter decided that one of the fictional personas' conversations was too spicy to continue and I had to carve out a custom policy for it. Run five stopped on a new Hermes quirk while I was away from the keyboard, and sat there with the meter running for most of the afternoon because the AI agent I had babysitting it was waiting for my approval before restarting. I had told it to. Lesson learned, rule changed.

Run six started at 9:24 that night. Six full restarts in one day. Still standing.

Somewhere in there I filed an issue upstream asking how the maintainers had handled these same Hermes behaviours in their own runs, since they must have hit them. That issue is still open with no reply, and you will notice that pattern comes up again.

The lights go out

By run nine the recording was airtight. Every Hermes behaviour captured as-is and documented, nothing rewritten. Ingestion is 13,539 messages at about forty seconds each, three personas in parallel, so about two and a half days, and it was finally going to be boring.

It was boring for three days. Then, on October 11th at 1:15 in the afternoon, the VM vanished.

Not crashed. Vanished. No error, no warning, just a machine that was there and then was not. I was out of cloud compute: the account runs on a monthly credit, the run had eaten it, and Azure switches everything off when that happens. The run was 82 percent ingested. The credit would reset on the first of the month. It was the eleventh. We had to pause the whole thing until October 28th.

Intermission

October 11th to the 28th. Seventeen days. I will spare you the internal debate about paying out of pocket to finish. I did not. Empirical kept shipping, a whole changelog's worth, because it is a product and it does not stop for a benchmark. I made peace with the fact that the back half of the ingestion would run on a server a month newer than the front half. Hermes and the runner are hash-pinned, so the agent could not get secretly smarter. If anything the memory got worse, because it had been through weeks of unrelated work in between. I decided to live with that rather than start over. I would make the same call again.

Empirical: Coming Home

On the 28th the credit came back and the VM came back and the run resumed. Then at twenty past midnight the server went unresponsive again under the same load, and I had to hard-restart the VM from the Azure console. Mongo came up muttering about an unclean shutdown, recovered from its last checkpoint, and every count checked out. That was the night I lost trust in a machine that could disappear.

So I brought it home. I stopped everything cleanly, copied the whole run, nine and a half gigabytes of it, to my desktop, and shut the VM down for good. The desktop is a Windows machine running Docker inside a Linux VM inside Windows, which is to say it is not a server, and it was already short on memory before I started. It held. The last 1,654 messages ingested at home.

The Closet Server

The ingestion finished. Now for the 600 tests, which need the memory server, the database, the runner, and a fresh agent container per test, all at once, on a machine that was already gasping.

I had an old eight-core AMD box in a closet. I put Ubuntu Server on it. It booted complaining that one of its drives was throwing errors, which turned out to be a dead disk on a bad cable that I unplugged. It could only see twelve of its sixteen gigabytes of RAM until I reseated the sticks. There is no option in its BIOS for any of this. None of it is interesting, all of it took hours. In the end the memory server and the database moved to the closet, the runner and the agents stayed on the desktop, and the two talked over an SSH tunnel because Docker on Windows cannot see the rest of the house.

It worked. It worked well enough to run the tests on for the next week.

365

November 3rd, the first full score. Fifth of six. Alex and Morgan were mid-pack, Riley was a disaster, 110 against the built-in memory's 152.

I was not thrilled with 365. But a full number is a thing you can look at, so we looked.

235 ways to be wrong

We went through every failed test by hand. Not to add rules for specific tests, which would be cheating and also pointless since the test set is held out from the agent. To find the kinds of failure. There were three.

First, the agent answered in chat when the test wanted an action in the app. The memory was there, the tool was there, it just wrote a nice paragraph instead of posting the comment.

Second, the agent searched its memory using the words of the request rather than the subject of the request, got vaguer results back, and wrote generalities. When I ran the same searches with the obvious subject, the needed fact was at rank one or two every time.

Third, the agent could see a standing preference in memory, something like "always send one message, not several", and followed the literal request anyway.

All three were habits, not recall. Did I mention that this version of Hermes, the one I am not allowed to change, is a pain to work with? The facts were in the store. So the change was not to retrieval at all. It was to how the agent is told to work with what it finds: look before you act, let the specifics you found carry into what you produce, and treat a recorded rule as outranking the wording of the request. That is the whole of it, and I am not going to pretend there was more.

I tested the change on the 60 failed tests that looked cheapest to fix. It passed 52 of them. Then I started run twelve, watched the first 45 tests, saw one more pattern, stopped it, reworked the wording and tested it again, and started run thirteen at 11:43 that night.

451

Alex 138, Morgan 163, Riley 150. First on the Hermes board by 27. Riley from 110 to 150. Thirteen runs, six weeks, blood, sweat, tears, a noticeably lighter wallet, and a screenshot I am going to keep.

Then I packaged the submission, and the real story started.

The zip doesn't fit

The official packager produced a submission of 1,874 MiB of JSON. The packager's own per-file limit is 512 MiB. The website's upload limit is 256 MiB. So I went looking for why.

The format stores, for every single message of every conversation, a full copy of the model's settings, including the complete text of every tool definition. Hermes has a long system prompt and its tool descriptions differ per persona and per profile, so nothing can be shared and it copies everything, tens of thousands of times over. I checked the other submissions to see how they got through. Graphon and AtMem, the two self-submitted rows above us, did not use Hermes. They wrote their own small harness: a fixed tool list, a short prompt, few tool outputs. Nothing varies, so nothing gets copied, and their zips came out small. Their runs cost a fraction of ours, which you can see in the leaderboard's own cost column.

I want to be honest about this, because it is the uncomfortable part. We could have done that. A harness we control would set the prompt, the tool list and the moment of retrieval exactly, and I am confident it would have scored higher and cost a fraction, because most of our misses were Hermes's habits rather than Empirical's recall. We did not do it because then our number would not sit next to Mem0's, Hindsight's or Honcho's. The comparison was the point. I wanted the row that says same agent, same model, same runner, different memory.

Then the other half of the answer, which arrived as a message from the agent doing the digging:

I read that and thought, FML. Six weeks of doing it exactly their way, and nobody had ever done it their way, including them. The maintainers' own Hermes rows never went through this packager. Their results were built by a separate script, straight from their internal run archives, and the script says so in its own description. Their docs for the participant path say, in so many words, that "a paid end-to-end run remains unverified". So the format and its limits had only ever been tested by small custom-harness runs. We are the first to push a full Hermes run through the official packager, and it is the format that broke, not the run. I was ready to pull my hair out.

The fix, and the form

The fix is small. Write each distinct settings object and system prompt once per phase and reference it from the rows. Nothing about any recorded message changes. It takes our zip from 652 MB to 93 MB, and the patched validator checks it at 451. I opened it as a pull request alongside the submission itself.

Then I uploaded the compact zip to the website, filled in the form, and got a receipt page:

Validation could not finish. The run remains pending for retry.

The website's validator is the old one. It cannot read a run that uses the fix, and it cannot hold a run that does not. Graphon, whose zip was small enough to fit the old format, hit the same wall for a different reason: theirs validated locally and the website's worker failed on it anyway. Their pull request, AtMem's, ours, and the fix are all open with no reply. The issue from September 28th is still open too.

I sent the Mem0 team a note with the three outcomes I would be happy with: merge the fix, take the full-size zip some other way, or tell me the shape they prefer and I will repackage. No hard feelings. A benchmark that takes off is a lot of inbox for a small team.

So here we are

Winners. Pending submission. Unverified by anyone but us, which is where you come in.

Check it yourself

Everything needed to check the number is public and none of it depends on us.

• The submission: mem0ai/dolphinbench pull 13, with the method note and a link to the release holding the evidence.
• The fix: mem0ai/dolphinbench pull 14, the validator change that lets a Hermes-sized run through.
• Two zips, same run. The full-format one is 652 MB and needs the per-file limit raised to read. The compact one is 93 MB and needs the fixed validator. Both hold all 600 test transcripts, every judge transcript, every app call, and the per-phase costs. SHA-256 for each is on the release page.

To check the compact zip, clone the fix branch and run the validator against it. No model calls, a few minutes:

git clone -b packaging-settings-tables https://github.com/mattgauzza/dolphinbench
cd dolphinbench && pip install -r requirements.txt
python -m harness.export_submission --release . --check /path/to/submission-compact.zip

The output ends with the pass count per persona and in total. If it does not say 451, I want to know. The submission README also describes how to rebuild the exact images and rerun the tests on the frozen ingestion, for very little in agent and judge calls.

What would help

If you work at Mem0, merging pull request 14 is the most useful thing you can do, for us and for whoever runs Hermes next. If you are anyone else who cares about memory benchmarks, a comment or a thumbs-up on that pull request tells them people are watching. The fix is small and changes nothing in any existing submission.

And to whoever designed this benchmark: I cannot tell if you are a sadist or a genius, but it was hell getting here. It is also the best benchmark out there for agent memory, which is the only reason I gave it six weeks instead of a weekend. Merge the fix, put the row up, and we are even.

And if you only want the lesson: your memory system is probably fine. Go look at what your agent does with what it finds.

All Empirical blog posts