An AI agent that learns from you is great. Who owns what it learns?
Empirical's Daydream emailed me an article about Hermes, an open source agent that learns skills from how you use it. It made me ask where a year of AI context actually lives, and what happens to it when you switch models.
This morning an email showed up that I did not ask for.
It came from Daydream, a feature of Empirical. It had looked at what I have been working on, found an article that connected to it, and sent it to me. No prompt, no search, nothing from me.
The article was Nous Research Secures a Massive $75M USD to Scale Its Hermes AI Agent on Hypebeast. Hermes is an open source AI agent, which is close to what I have been building around, so it was a good hit.
But what stuck with me was one line about how the agent works.
The line that got me
Hypebeast describes it like this:
"Unlike static language models, Hermes Agent automatically analyzes user patterns to acquire and refine new skills without manual developer intervention."
Read that again. The agent learns from how you use it. It gets better at being your assistant the longer you work with it.
That is a genuinely good feature. It also raises a question I had not asked in quite this way before: who owns what it learns?
A few more facts from the same article, so we are on the same page. Nous Research is reportedly raising $75 million at a $1.5 billion valuation. The agent ships with "built-in capabilities for web searching, coding and image analysis." And its GitHub repository has "over 214,000 stars and nearly 40,000 forks."
That is a lot of people betting on an open source agent, and I think that is good news. My point is narrower. Open source tells you who can read the code. It does not by itself tell you where your accumulated context ends up, or how easily you can take it with you.
A year of working with an assistant
Imagine you spend twelve months with one assistant, one that learns. Along the way it picks up:
- how you like code structured
- which decisions you made, and why
- the names of your projects, clients and collaborators
- what you already tried and rejected
- how you write, and what annoys you
That is arguably the most valuable thing the assistant has, more than the model underneath it.
Now suppose you decide you like another model better. A new release lands, a competitor is faster, or your team standardizes on something else.
What happens to that year?
Three places context can live
Most setups end up in one of these.
| Where it lives | You control it? | Survives switching tools? | Other tools can read it? |
|---|---|---|---|
| Inside the assistant product | Mostly no | No | No |
| In local notes and files | Yes | Yes, if you copy them | Only if you paste them in |
| In a layer you own, reachable by any assistant | Yes | Yes | Yes |
The first row is the default. Memory is a feature of the product, so it is tied to the product. That is not a conspiracy. It is how the incentives line up: a product that knows you well is a product you are less likely to leave.
The second row is honest and sturdy, but manual. Notes you have to paste into every new chat are notes you will eventually stop pasting.
The third row is the one I wanted.
What "somewhere I control" looks like
This is the problem Empirical is built for:
- Your memory is a graph of things you chose to save: decisions, preferences, plans, lessons.
- Any assistant that speaks MCP can read and write it. That includes ChatGPT, Claude, and a CLI.
- When you switch models, the memory does not move, because it was never inside the model.
Here is the shape of it, as an illustrative example:
You: Remind me why we picked Postgres over Mongo for this project.
Agent: (queries your memory first)
You decided on 3 March: relational reporting needs, and you
wanted transactions across billing and usage. Mongo was
rejected for that reason.The answer came from your memory, not from the assistant's. Swap the assistant tomorrow and the same question gets the same answer.
"Couldn't I just do this myself?"
Mostly, yes. A folder of markdown notes that you paste into each new chat is the second row of the table, and it works. If you are disciplined about it and only use one or two tools, it may be all you need.
What changes with a shared memory layer is the part that does not scale by hand: the agent can look things up itself instead of waiting for you to paste, every tool you use reads and writes the same place, and you are not the integration layer between your own assistants.
That is the real difference. Not smarter search, but persistence across sessions and tools without you rebuilding it each time.
An agent that learns, plus memory you own
I do not think these two ideas are in conflict. A self-improving agent like Hermes is useful, and they stack:
- The agent gets better at doing the work.
- Your memory layer holds the durable facts about you and your projects, in a place you can read, edit and delete.
If you ever change agents, you keep the second part and only rebuild the first.
Where Daydream fits
Back to the email that started this.
Most memory systems are passive. You ask, they answer. Daydream goes the other way: it looks at what is in your memory and brings something back to you without being asked.
Today that was an article about an open source agent, which landed right on what I am working on. Nobody told it to find that. It did it because my memory said that was where my head was.
I want to be honest about the limits. It is early. It does not always hit, and some days the article is only loosely related. The first version of this idea felt like a mirror, repeating me back to me. The goal is closer to a friend who says "you should see this."
A portability test you can run today
You do not need Empirical for this. Use whatever assistant you have now.
- Ask it: "What do you know about how I work?"
- Read the answer. Is it accurate? Is it complete?
- Now ask yourself: could I export this in a format another tool could read?
- If the answer is no, that is your lock-in, measured.
The point is not to panic. Some lock-in is a fair trade for a good product. The point is to make the trade on purpose.
What I take from this
Agents are getting better at learning from us. Good. That makes the context they collect more valuable every month, and the question of who holds it matters more each year, not less.
I would rather hold it myself. That is why I built Empirical.
If you want to try it, you can start here. And if you run the portability test above, I would like to hear what you found.