I Tried Building a Personal Brain. The Version That Finally Worked Was Just Markdown.
This entire thing may look like it was written by AI. And to be fair, parts of it were helped by AI. But what I have built here is not just an AI-generated blog post. It is the output of a system I built for myself: a Custom Wiki, inspired by Andrew Karpathy’s LLM wiki idea, that has kept track of the work I have done on this topic since the beginning of the year.
It has tracked the iterations, the hackathon projects, the failed attempts, the graph databases, the embedding databases, the production versions, the coding-agent experiments, the voice memos, the ChatGPT conversations, the costs, the storage sizes, and the final architecture I have now landed on.
After six or seven months of trying different versions of this idea, I think I have finally found the version that makes the most sense for normal people to use.
The problem I kept returning to
The broad idea was always the same: make messy human data searchable and usable by meaning. Not just searchable in the keyword sense, but searchable in the “find the thing I mean” sense.
At first this showed up as search across images and videos. Then it showed up as search across raw text and SQL. Then it became graph databases, agents, and SaaS interfaces. Eventually it became a personal brain: a system that could remember my conversations, notes, transcripts, projects, mistakes, and decisions in a way a coding agent could actually use.
The image/frame-search version
One of the earliest versions was an image-frame search engine. The idea was simple: take video or image frames, describe what is happening in each frame, embed those descriptions, and let someone search for the exact frame or moment they wanted.
I pitched a version of this at a hackathon in Hyderabad for the cinema field. It was extremely last-minute. I made up my mind that I wanted to participate at around 9 PM, and by 10 PM I was on an overnight bus to Hyderabad.
That version mattered because it showed me that the problem was not limited to text. The same idea applied to images, videos, frames, transcripts, conversations, and eventually a full personal brain.
The ISB IVI SaaS pitch
There was also a SaaS version of this idea that I pitched to ISB’s IVI program in December. I pitched it to them in Bangalore and was invited to Hyderabad for the final pitch. Unfortunately, I did not get selected to go further, but that pitch was another important checkpoint in the journey.
That version was closer to a hosted product: a polished interface around the personal-brain idea, with the expectation that users would subscribe to it and use it through a web app.
The pitch helped me see the gap between what looked good as a product and what actually made sense for users over the long term. A hosted SaaS version was convenient, but the moment I thought about costs, privacy, latency, and data ownership, it started to feel like the wrong abstraction.
The Atimate Code SQL version
Another important milestone was the Atimate Code hackathon in HSR, where I built the SQL-backed version of this idea. In that version, raw text lived in SQL and the coding agent could query it through a structured SQL interface.
That version worked beautifully with their coding agent because the agent did not have to guess where memory was. It had a clear interface: ask the database, retrieve the relevant rows, and reason from there. I ended up placing second at that hackathon.
The SQL version taught me that agent memory does not need to be magical. It needs to be structured, inspectable, and easy for the agent to query.
A video of me pitching it
There is also a video from this journey of me pitching the idea. I wanted to include it because the project was not just something I thought about privately; I kept trying to explain it in public, at hackathons and pitch settings, and each pitch forced the idea to become clearer.
Video of me pitching the personal-brain/search idea during the journey.
The graph database phase
After that, the versions got more complicated. I built and explored more production-like systems: engines connected to graph databases, Neo4j-style architectures, GraphRAG ideas, and agentic retrieval systems.
These systems were technically interesting. They made sense if the goal was to build a full production memory engine. But they also made the product heavier. More infrastructure. More moving parts. More cost. More debugging. Less malleability.
What I started seeing in the market
The more I built, the more I started noticing something important: people did not just want “AI memory.” They wanted memory they could control.
- They wanted it to be malleable.
- They wanted to edit it.
- They wanted to understand what was happening.
- Some people wanted local LLMs.
- A lot of people cared about privacy.
- They wanted their data stored locally.
The more complicated my system became, the more it moved away from what people actually seemed to want.
Why the hosted SaaS version did not make sense
At one point, I tried building this as a web product: raw text in SQL, an agent querying it, a coding-agent SDK, a Next.js interface, and a SaaS subscription model.
Technically, it worked. But the unit economics did not. If I wanted it to be fast, reliable, and hosted for users, I had to run it through a proper SDK and deploy it on something like Azure or Vercel. That made the cost structure ugly quickly.
The product was supposed to feel personal, private, and lightweight. But the infrastructure was becoming expensive, centralized, and heavy. That felt wrong.
What did not work
Native search
Native search was cheap and useful. It could index my ChatGPT conversations and help me find files. But it was shallow.
It could tell me where a term appeared. It could help me locate a conversation. But it could not reliably tell me what mistakes I had made across LRDI sets, which projects were connected, how one idea evolved over months, what I had tried and abandoned, or what was worth writing about.
It was search, not memory.
Cognee / Cogni-style memory
Cognee was much more powerful. It had graph extraction, embeddings, summaries, chunks, retrieval, and graph traversal. It taught me a lot about what a graph/RAG memory pipeline looks like.
But for my use case, it was too heavy. It needed an LLM API, an embeddings API, dataset management, runtime complexity, API debugging, and a much larger storage footprint. It was good technology, but it did not match the version of the product I wanted: something local-first, inspectable, and easy to modify.
What finally worked
The version that finally worked was much simpler.
raw files → LLM-generated Markdown wiki → deterministic JSON index/graph → coding-agent skillI call it my Custom Wiki. The raw data stays untouched in a files/ directory. That includes ChatGPT conversations, coding-agent sessions, voice memo transcripts, notes, work files, and anything else I add later.
Then an ingestion script uses a cheap nano model to summarize those raw files into structured Markdown pages. Each page has metadata like tags, projects, mistakes, source paths, tools, concepts, and aliases. Deterministic scripts then build a JSONL index, a graph JSON file, and a manifest.
custom_wiki_index.jsonl
custom_wiki_graph.json
custom_wiki_manifest.jsonSo the coding agent does not need to “remember” everything magically. It can query the index, inspect the graph, open the Markdown page, and then go back to the raw source if needed.
The LLM is not the database. The LLM just turns messy raw files into clean Markdown. The actual memory is local, inspectable, and structured.
The cost surprised me
I ingested around six months of my ChatGPT history, work files, notes, voice memo transcripts, and coding-agent work into this Custom Wiki. The total cost was around ₹74–75.
510 sources
1,051,170 tokens
$0.769
₹74.34That is what makes this interesting from a bring-your-own-key perspective. Instead of charging people monthly for a hosted memory system, this can run locally. If they want bulk ingestion, they can use their own API key. If they already use a coding-agent subscription, the querying and editing workflow can happen through that.
The end goal
The end goal is not just to build “AI search.” The end goal is to build a personal knowledge system that compounds.
A system where every conversation, note, transcript, and project becomes part of a local memory layer that I can actually use.
- I should be able to ask: What have I done on this idea since January?
- I should be able to ask: What mistakes do I keep making in LRDI?
- I should be able to ask: Help me write a blog about my journey building a personal brain.
And the system should pull together my hackathons, failed attempts, costs, architectures, and conclusions.
That is what finally started working. Not because I built the most advanced graph memory system, but because I stopped trying to make memory magical and started making it readable.