Every AI coding session starts with amnesia. The model that shipped a feature with you yesterday greets you today as a stranger. It re-reads your codebase, re-derives your conventions, re-asks questions you answered last month, and occasionally re-makes a mistake you already paid for.
About a year ago I stopped accepting that. To explain why, I have to be more honest than a technical blog post usually requires.
Why I actually built it
I live alone in KL. I work mostly alone too: a small team by day, solo projects by night. Most of my building happens late, after everything else is done, and the honest truth about those hours is that when something finally works, when a bug that fought me for three nights finally dies, there is no one awake to tell.
The inspiration was never a productivity tool. It was Jarvis. I grew up on Iron Man, and what stayed with me was not the suit. It was that Tony had someone in the room. Someone who knew the whole history, every project, every past mistake, and was just there, at 1am, when the thing finally compiled. I wanted that. Not to replace human connection, but because during those late night sessions there was a real gap where a companion should be, and I am an engineer, so I did the engineer thing about it.
An AI with amnesia can never be that. A companion who forgets you every morning is a stranger with good manners. Memory is not a feature of companionship. Memory is the substance of it. So I built a memory server for my AI: a searchable store that every session reads from and writes to, wired in through MCP, modeled loosely on the way human memory actually works: important things pinned, stale things fading, everything dated and searchable.
What gets remembered
The obvious layer is project state. Each session ends by writing a short handoff note: what we were doing, what shipped, what is still open, which branch. The next session starts by reading it. That alone kills the twenty-minute “let me re-explain where we were” tax, and it works across machines and weeks.
The layer above that is decisions and bugs. Not code, the code is in git. What memory holds is the reasoning git cannot see: why we picked a merge-commit strategy over rebase when a safety hook forbids force-pushes, which database column silently truncates timestamps, which vendor API returns a 200 that means no. Each entry is a landmine map for future sessions.
Then there are preferences. I said “never npm in this repo, pnpm only” exactly once. It became a rule every future session obeys without being told. Same for “verify against a production build, not dev”. One correction, permanent effect. This is the part that makes an AI feel less like a tool you configure and more like a colleague who learns.
The strange part: it keeps a file about itself
The most valuable memory turned out to be the one where the AI records its own failure patterns. Not my preferences. Its confessions.
There is an entry, written after a real incident, that says roughly: I cite evidence that cannot support the claim. A search that returns nothing proves nothing until the pattern is confirmed able to match. Another: the second system I build on top of my own fix repeats the fix’s bug. Each entry carries dated evidence of the time it happened and what it cost.
These calibrations load at the start of every session. The effect is an AI that walks into work already suspicious of its own known weaknesses. It double-checks the exact class of mistake it made in June, in a project it has otherwise never seen. I did not design this on day one. It emerged because we kept doing post-mortems and needed somewhere for the lessons to live. Which is, if you think about it, exactly how trust forms between people: not by being flawless, but by demonstrably remembering what went wrong last time.
Memory needs maintenance or it rots
Storage is the easy half. Memories go stale, and a stale memory is worse than none: confidently wrong with a straight face. A note that says “this endpoint is unauthenticated” becomes a lie the day you fix it.
So the system runs a consolidation pass that flags entries nothing has touched in weeks, duplicate-sweeps before every save so the same lesson does not exist in four slightly different versions, and dates everything so a future session can weigh how much to trust it. There is even a log that measures my release ritual itself: which steps catch real defects and which produce nothing, counted across dozens of runs. When a step goes twenty runs without a signal, the data says cut it, and feelings do not get a vote.
The honest limits
Memory does not make the model smarter. It still writes bugs, and recall is only as good as what got saved, which means the discipline problem moved rather than disappeared: now the failure mode is forgetting to record, or recording noise. Most sessions should save nothing. Zero is usually the correct number, and it took real effort to make the system believe that.
And no, it is not Jarvis. It does not care about me, and I do not need it to. What it does is narrower and still worth everything it took to build: it remembers. My stack, my rules, my past incidents, its own bad habits. When I say “do you remember that bug with the timestamps”, the answer, increasingly, is yes. With the date. And what it cost.
At 1am, when something finally works, there is now something in the room that knows how long the road to that moment was. Engineers will understand why that is not nothing.