Giving Claude a Second Brain

Every conversation I have with Claude starts the same way: with Claude knowing nothing about me, nothing about my homelab, nothing about any of the projects I'm working on. I fix this the obvious way, by pasting context, pointing at files, or re-explaining decisions I've already explained fifty times.

That works, and it costs more than it looks. Everything I paste lands in the context window, and the context window is finite. On a standard plan it is finite enough to feel, which changes the problem entirely: the tokens spent explaining the background are the same tokens the actual work needs. Front-load enough history to make a session useful and a real portion of the budget is gone before anything has been done. Run long enough and the conversation gets compacted, the summary quietly loses the detail that was pasted in, and it gets pasted again. I have had sessions where the first ten minutes were me re-explaining my own infrastructure, and the first thing lost to compaction was that same explanation.

The deeper problem is that this is the wrong shape. Handing over everything that might be relevant, up front, on the off-chance, is the opposite of how retrieval should work. What I want is for the model to pull the two or three things that actually bear on the conversation at hand, at the moment it needs them, from something that already knows them. Then the cost scales with relevance rather than with the size of my history. The model is capable; the context it gets handed is threadbare, and it is threadbare because the only mechanism for filling it is expensive at both ends.

I have a lot of working contexts, each with its own repo, its own vocabulary, and in several cases its own rules about what can be discussed where.

This post is about a system I built to do exactly that, and have now been running for a few months. I'm calling it claude-brain. Some of what follows is the design as I planned it; some of it is what the design became once it met real use, and that was not the same thing. On a personal project, what changes the plan is you, a few weeks later, using what you built.

What separation has to mean

When I sit down to work on the novel, I want Claude to remember what I decided about a character three chapters ago. When I open the homelab, I want it to know which services I run and which decisions I've already taken about my network. For a blog post, I want the standing methodology from the last three posts on the same topic, so I'm not reinventing my own voice every time.

What I emphatically do not want is for any of those contexts to bleed into another one. The novel's plotting should not leak into my technical writing. The homelab's secrets should not turn up in a book draft. Different projects have different rules about what belongs in them, and the right place to enforce those rules is the system itself, not my own vigilance at three in the morning.

A discipline-based solution ("just remember not to mix them") fails on a long enough timeline. I forget things, and Claude hallucinates, so a system that depends on either of us staying vigilant will eventually let something through. A system that makes the correct direction the easy direction, and the wrong direction outright impossible, will not.

Prior art: Karpathy's LLM wiki pattern

The starting point was Andrej Karpathy's short gist describing a pattern he calls the LLM Wiki. The sketch is three layers:

1. Raw sources, immutable. Meeting notes, logs, papers, transcripts, chat history.
2. An LLM-maintained wiki of interlinked markdown pages, compiled from those sources by the model itself.
3. A schema document that describes the structural conventions the wiki is supposed to follow.

And three operations: ingest (convert raw sources into wiki updates), query (answer questions from the wiki, not from the raw sources), and lint (periodically clean up contradictions, duplicates, and stale links).

The key move is treating the wiki as a distillation of the raw material, not a replacement for it. The raw stuff never gets edited, so there's always a ground truth. The wiki is the index the model actually reads, and the model is the one maintaining it.

This is right, and I've adopted it almost wholesale. But I've made one significant change, and it is easier to see side by side:

That right-hand column is the design I started from, not the system I ended up with. Two of those rows changed once I actually used the system.

Where claude-brain diverges: the knowledge layer isn't for humans

Karpathy's wiki is a folder full of markdown, readable by humans as a side benefit. My knowledge layer is a Postgres database, and it is explicitly not shaped for human reading. It is shaped for Claude.

The reason is that I already have an enormous amount of human-readable content. Blog posts. Book drafts. Homelab documentation. Novel chapters. These are the raw sources. Producing another human-readable layer on top of them would just create a second thing to maintain. What I actually need is a distilled, structured index that Claude can query cheaply and reason over quickly. That's not a wiki; that's a database.

There's a second reason, and in daily use it has turned out to matter more than the first. A folder of markdown lives on a machine. Whichever machine that is becomes the place the knowledge is, and every other device is a second-class citizen: you sync it, or you clone it, or you go without. A database on the network belongs to no device in particular. It sits on the homelab and anything that can reach it gets the same answers.

That distinction shows up constantly. Claude Desktop on the Mac and Claude Code in a terminal are talking to the same store, not to two copies that have quietly diverged. If I work from a different machine, there is nothing to synchronise first. A card written in a terminal session at lunchtime is there in a desktop conversation that evening, because there was only ever one copy of it.

The caveat is that this is not free or automatic. Every app has to be configured to point at the MCP endpoint, and each has to be pointed at the right subject. What the database buys me is that once a device is configured, it is genuinely equivalent to every other device rather than approximately equivalent. There's no sync to have failed and no local copy to be stale. The knowledge is somewhere, singular, and everything else just reads from it.

So the claude-brain schema has two core entity types:

• Topic cards, which describe what something is. A methodology, a tool, a person, a standing decision, a character, a location. Small, atomic, updatable.
• Episodes, which describe what happened. A work session, an incident, a deployment, a decision being taken, a scene being written.

Everything Claude knows gets modelled as either a thing or an event. Tags, links, and source pointers live in join tables so the graph can be traversed. Since 2 September 2026 the schema has also gained source provenance and a separate documents store, alongside the topic and episode tables.

Retrieval started as Postgres full-text search (tsvector) and nothing else. That was a deliberate choice to avoid the complexity of a vector store on day one, but I kept the MCP tool surface agnostic about how search actually worked, so it could change underneath without anything above it noticing. Search has since changed. It is now hybrid: full-text alongside pgvector embeddings generated on write by a small embedding model running locally through Ollama. Every card in the brain is embedded. Nothing above the search function changed when that landed. That is the whole argument for putting the seam there in the first place.

The rules engine

The rules engine sitting on top of the database is what makes this work, and that engine has three jobs: stop the wrong things being written, put the right things in the right place, and keep the brain clean over time.

Subjects and Postgres roles

The top-level concept is a subject. A subject is a namespace with its own access rules, and each body of work I keep separate gets its own subject. Throughout this post I'll use a shared personal subject plus one each for the homelab, the novel and the book. The labels are illustrative; the structure holds whatever you put in it. Subjects are enforced at the database layer, as Postgres schemas, not in application code, with one Postgres role per subject. A role has full access to its own schema and no access at all to any other subject's schema.

The MCP server opens its connection as the role matching the subject the session was launched with. A write aimed at the wrong schema fails at the wire, and no check in my own code is involved.

The whole isolation model is about a dozen lines of SQL. Read them rather than take my word for it:

-- A project role owns its own schema outright.
GRANT USAGE ON SCHEMA homelab TO role_homelab;
GRANT SELECT, INSERT, UPDATE, DELETE
    ON ALL TABLES IN SCHEMA homelab TO role_homelab;
 
-- ...and reaches the shared schema, deliberately, including writes.
-- Note the absence of DELETE: an upcast can add and amend, never remove.
GRANT USAGE ON SCHEMA personal TO role_homelab;
GRANT SELECT, INSERT, UPDATE
    ON ALL TABLES IN SCHEMA personal TO role_homelab;
 
-- No GRANT to novel, book, or any other subject's schema.
-- The isolation is the statement that isn't there.

Sequence grants and default privileges are left out for brevity, so this is not a working bootstrap script: paste it as-is and your first insert fails on a sequence permission. Both omissions fail closed rather than open, which is the right direction for a mistake to run in.

That last comment is the entire security model. There is no policy engine deciding what a session may touch, no policy file, no middleware. A role can reach what it has been granted and nothing else, and the grants for the other subjects were simply never written.

The asymmetry is the point, so it pays to be precise about which direction is sealed. The personal role has grants on the personal schema and nothing else. It cannot read any project schema, at all, ever. That is the property I actually care about: nothing from the novel can surface in a homelab session, and no amount of clever prompting will get it there, because the data is not visible to the connection.

The other direction is deliberately open. A project role can read the personal schema, so reusable methodology flows down into whatever I'm working on automatically. It can also write to it. The first version of this design, the one in my notes, said the boundary was sealed in both directions. It was not, and I would have told you it was. It's called an upcast: something learnt inside one project that turns out to be generic, a Postgres gotcha found while building the brain itself, or a backup pattern worked out for the homelab, promoted into the shared space so the next project inherits it. That has to be possible, or the shared brain never learns anything from the work that actually happens.

What stops an upcast becoming a leak is two things layered on top of the role model, because the grant on its own would let it through. A deny-list scans every write into the personal schema for tokens that tie it to one specific project, and for PII-shaped patterns, and refuses on a hit; it is deliberately noisy, because a false positive just means the card stays in the project schema instead, which is the safe outcome. And an outbox table records every write with an actor, an operation, and a reason, so an upcast without a justification is a flag rather than a silent event.

The honest version is that the direction that must never happen is structurally impossible, and the direction that should be rare is possible, guarded, and audited. I don't have to remember to stay inside the lines going one way. Going the other way, I have to say why.

The kind vocabulary

Every topic and every episode has a kind. The list of valid kinds lives in a single YAML file, loaded by the server at startup and consulted by every write. Unknown kinds are rejected outright with an error listing the valid ones.

The rule I set for myself was that every kind is either a thing or an event, never both. The same distinction runs through the technical work and the fiction. Seeing it hold in both is what convinced me the split was real rather than convenient:

Read the pairs across and the discipline shows up. decision is a topic, the standing decision itself; decision-made is the episode where it was taken. Same split for the fiction side: chapter-summary is what the chapter is about, chapter-write is the session where it was drafted. A bug is always an episode, because bugs happen at a moment in time; a recurring class of bug is a design pattern, filed as a topic.

Having a single authoritative file forces discipline. Claude cannot quietly invent a new kind on the fly; if a new category is needed, I edit the YAML. That rigidity is deliberate. Without it, a year from now I'd have seventeen near-synonyms for "thing I once wrote down" and no way to query reliably.

Search before add

The most important rule in the curation loop is search before add: before creating any new topic, Claude must search the brain for existing matches. A confident match updates the existing topic rather than duplicating it. Loosely related matches get a new topic with mentions links back to each of them. Only when the search returns nothing meaningful does Claude create a fresh island topic.

This is the single most effective quality control I've added. Without it, every session would re-create the same methodology card from scratch, with slight variations, and within a week the brain would be a lumpy sediment of near-duplicates. With it, topics accumulate information over time rather than piling up sideways.

Capture at session end, and why I turned it off

The original design had this fully automated: a SessionEnd hook fired at the close of every Claude Code session, ran a headless subtask that read the transcript, and wrote topics and episodes without my involvement. It printed a one-line receipt so I could see what it had done.

I built it, ran it from April until the start of July, and then retired it. The reasoning generalises, so here is why.

The problem with a headless job that writes to a database is that it can fail quietly. Mine did. A routing bug meant a run of sessions distilled into the wrong place, and because nobody was watching a process designed not to need watching, it ran that way for most of the time it was live. The isolation guards held throughout. That is the one genuinely reassuring part of the story. The cost was history: I recovered the most recent month by re-running the distillation by hand against the transcripts still on disk, and lost the two months before that outright, because Claude Code prunes transcripts after about thirty days.

So capture is now deliberate. A /session-close command sweeps the session, proposes a set of subject-tagged cards against an explicit rubric for what is worth keeping, and writes nothing until I confirm. What I see at the end of a session looks like this:

Proposed for homelab:
  + episode  deploy          "Rebuilt brain stack on pgvector image"
  + topic    environment     "embeddings host, nomic-embed-text, 768-dim"
 
Proposed for personal (upcast):
  + topic    design-pattern  "Structural enforcement beats procedural"
      reason: generic, no homelab specifics
  ~ topic    tool            "psycopg3"  (append: search_path scoping)
      reason: applies to any psycopg3 work
 
  4 proposed, 2 upcasts. Write these? [y/n/edit]

The ~ is an update to a card that already exists: the search-before-add rule doing its job rather than creating a fifth near-duplicate of the same thought. It is more friction and I think the friction is correct: every write is now something I have seen, with a subject I have agreed, and there is no unattended model run that can drift without anyone noticing.

The trade is real and I'll state it plainly. If I don't run /session-close, nothing is captured. I chose deliberate over silent, having been burnt by silent.

The lint loop

A brain_lint pass walks a subject looking for three things curation got wrong: orphan topics, dead links, and unexplained upcasts. Dead links get removed automatically when I ask it to apply; orphans and upcasts are only ever reported for me to look at. Duplicate detection and contradiction detection are planned, not built. This is Karpathy's lint operation applied to a structured store rather than a markdown tree.

It runs on demand rather than on a schedule. I keep meaning to put it on a timer and keep not doing it, partly because the same argument that retired the capture hook applies here too: an unattended job that rewrites the brain is exactly the kind of thing I'd want to be watching.

How it's actually deployed

The rules engine is the interesting part, but none of it matters if the thing falls over or gets rebuilt inconsistently, so the deployment story matters too. Claude-brain first ran on a Proxmox LXC in my homelab. Caddy fronted it, an Ansible role brought it up, and dnsmasq on the router resolved its name from the same network configuration repo as every other internal name, with no public DNS record anywhere. On 1 September 2026 I moved it to run locally on my Mac. Laid out, the current shape is unremarkable, which is the point:

Postgres and the MCP servers run as a Docker Compose stack, now under colima on my Mac rather than on a homelab LXC. One container for Postgres, and then one brain-mcp container per subject, each written in Python (FastMCP + psycopg3), and each bound to loopback only. That last detail is not incidental: each container is handed exactly one subject and exactly one role's credentials, so the subject boundary is a process boundary as well as a database one. A container physically does not hold the credentials to reach another subject's schema. The compose file, Dockerfile, migrations, and server code all live in a single repo alongside the schema design and this post's raw notes.

Bringing the stack up is now one script, not an Ansible role. brain-up.sh starts colima if it isn't already running, checks that Ollama is answering on 127.0.0.1:11434 and warns if it isn't, because a search would otherwise fall back to keyword-only without telling me, then brings the compose stack up and health-checks each of the four servers in turn. Schema bootstrap is a separate step: a fresh host needs db/bootstrap.py run by hand, not something the start script does for it. There's nothing in front of the servers, no Caddy and no bearer check, because there's nothing to front: each one is reachable only from the Mac itself, on loopback.

The rules engine only holds if the deployment does. A careful schema design and a locked-down role model are worth nothing if the next deploy overwrites the Postgres container with a fresh volume and loses the lot. The same config-as-code discipline I apply to endpoint management matters here too: what rebuilds the brain is a compose file and a script committed to the repo, not a step I remember to run by hand.

What it feels like to use

The change is small in any individual session, but compounding. When I ask Claude in one project's context "have we seen this pattern before", it answers from the personal brain, because the session has read access to the shared space. When I ask the same question in a different project, I get that project's knowledge plus the shared space, and nothing from anywhere else. When I close out a chapter of the novel and the episode gets recorded, the next time I want to ask "what was I doing last time in this storyline", there's an answer. None of that context had to be handed over by me at the start of the session. It was already there.

What I notice most is how much less I'm pasting. Claude knows things. Not everything, not yet, but enough that the baseline starting state of a conversation is no longer zero. That changes the economics of a session in the way I hoped it would: the window fills with the work rather than with the preamble, and a few retrieved cards cost a fraction of what dumping the background used to. The sessions that used to run out of room now mostly don't.

What I learnt building it

Before the lessons, here's the scoreboard, separating what changed cheaply from what changed expensively:

The top two cost me an afternoon. The next two cost me months, and one of them cost me data I cannot get back. That gap is the whole lesson, and three smaller ones follow from it.

First, structural enforcement beats procedural enforcement every single time. I initially considered doing subject isolation in the MCP server's application code. Moving the boundary down to Postgres roles made the direction that matters impossible rather than merely discouraged. A database enforces a rule; a program only suggests one. That is not a claim about Postgres. It is the same argument as enforcing access in the directory rather than in the application: ask of any control you have built whether the wrong action is impossible or merely discouraged. The corollary is that you should be honest with yourself about which rules you have actually enforced and which ones you have only written down. My first draft of this design said the boundary was sealed in both directions. It was not, and a review caught it before any code existed. That is the cheapest possible moment to find out.

Second, a rigid vocabulary is worth its inflexibility. The temptation with LLMs is to let them be flexible because they can. Resisting that temptation, pinning the vocabulary to a YAML file that the model cannot bypass, is what stops the brain becoming mush over time.

Third, I had this one backwards. I originally concluded that automation only works if it's cheap and visible, and that the automatic curator plus a one-line receipt had struck the right balance. What actually happened is that the automatic half failed quietly and the receipt was not enough to catch it, because a receipt only tells you what a process thinks it did. The lesson I'd write now is narrower: automate the retrieval, because being wrong there costs you a bad answer you can see. Be far more careful about automating the writes, because being wrong there costs you a corrupted brain you cannot see.

What's next

The brain is still internal only, reachable from my Mac and nowhere else. Exposing it externally via Cloudflare Tunnel, with proper MCP remote-server authentication so Claude on mobile and the web app can reach it, remains the plan and remains undone. I want to be able to ask "what did we decide about X" from my phone, on the road, away from the laptop, and get the same answer I would at my desk. It keeps losing to work that was more urgent, which is the honest reason most roadmap items slip.

The other outstanding piece is a human-readable view over the same Postgres store, not to replace the brain but to let me browse what has accumulated and see the health of it: duplicate clusters, orphans, subjects that have gone quiet. That one is designed and not built.

The code isn't public, and I'm undecided about whether it should be. The interesting part is the shape rather than my particular implementation of it, and the shape is entirely described above.

The brain started empty. It grew by use. That was exactly what I hoped for, and slower than I expected. My phrase for it during the design was "training it", and that still captures how it feels: each session leaves a slightly smarter version behind. The next conversation inherits that. And the one after that. What I'd add, a few months in, is that the compounding is real and it is not free. The system needs tending, and the rules need to hold up once they meet real use. The parts I was proudest of designing were not the parts that turned out to matter. Claude no longer starts every session from zero. That was what I wanted. Getting there just involved being wrong about more of the details than I expected.