The Weights Kept Nothing

I trained a small language model on my entire personal knowledge base. 8,281 files, 97,437 chunks, a night of pipeline work. Then I asked it 131 questions whose answers were certainly in my notes, and it answered exactly zero of them correctly. Not one. Even a second model judging semantic equivalence agreed: zero.

I count this as the cleanest result I have gotten from an experiment in a long time, and the reason is the surprise that came after it.

Why I built an oracle

Every inference starts from zero. A model has no memory of the conversation it had yesterday and no memory of the notes I have been writing for years. Tools help. I keep a search index over my notes, my agents consult it, and it works well enough that I stopped noticing it. But retrieval is work. Every answer means a query, a rank, a rerank, context assembled by hand. It felt like the model was reading my notes over my shoulder instead of knowing them.

So the idea: train a small model on the whole corpus until the facts live in the weights. An oracle. No search step, no context assembly, just a question and an answer from memory. The literature warned me this does not work. A documented experiment with the same design reached 7 percent accuracy where retrieval-based memory reached more than 40 percent [1]. I ran mine anyway. Experiments exist to test what you doubt, and the setup was cheap: a local two-billion-parameter model, LoRA adapters, one evening of pipeline code, my own Mac as the training machine.

The setup was stricter than the training

I wrote 150 test questions first, before any training, and froze them. Single facts, cross-file chains, dated facts, abbreviations, and questions my notes cannot answer at all, the last ones to measure whether the model knows what it does not know. Every gold answer pointed to its source file, and I locked every source file out of the training data, so the test could not leak into the weights.

Then I measured the thing I was trying to beat: my existing search. Given the top five results, it surfaced the correct source for 73 percent of the answerable questions. That was the bar.

The weights kept nothing

After 600 iterations over the full corpus, the model scored zero out of 131. Its answers were fluent and confidently wrong. Asked what an abbreviation meant, it explained that the letters stand for something close but invented. Asked who a family member is, it delivered a polite essay about the name being common. Zero is below the published baseline for this kind of experiment, and it is also the honest number, re-measured after I fixed a bug in my own evaluation pipeline first.

One thing did transfer into the weights: behavior. The model learned to say “I don’t know.” Asked 19 questions that my notes cannot answer, it correctly declined 10 of them. That is the most interesting result of the whole experiment. Behavior is trainable. Facts are not. The second variant, trained on distilled question-answer pairs instead of raw text, answered more fluently and believed more nonsense: its decline rate fell to 4 of 19. Better style, worse honesty.

The fix was a search index, not a model

While I was debugging the baseline, I found something strange. My main instruction file and 276 curated memory files were not in the search index at all. A symlink had never been followed, and an exclusion pattern had silently swallowed a whole directory. Every agent in my setup had been searching without my most important files for months.

Fixing the index, no training involved, lifted retrieval quality from 29 percent to 53 percent, measured the same way as before. And once I also counted answers that were merely retrievable from the top five results rather than surfacing their exact source file, the number was 73 percent. The honest version: the metric changed slightly between the two measurements, and the direction did not.

The training had learned nothing. The cleanup had changed everything. I spent a night trying to rebuild my brain in silicon, and the actual memory, the notes, had a hole in the drawer.

What I would do instead

The field has settled this question, and the answer is less romantic than an oracle: behavior belongs in weights, knowledge belongs in context. Keep the volatile facts in a store you can update, delete, and timestamp. Use fine-tuning for the stable things, tone, format, judgment [5]. The leading open memory layers reach 92 to 95 percent on the reference benchmarks [2], and my unglamorous search reaches 73. Fine-tuned weights reach zero. Retrieval is not a compromise. It is the technology [4,6].

Three fixes would help my setup more than any training run. A threshold that says “no confident answer in the notes” instead of silently returning the top five, because that silence is how retrieval lies. Chunk context, so every snippet knows which document it came from, a technique Anthropic documented with a 49 percent reduction in retrieval failures [3]. An alias layer, because my notes speak in abbreviations and my questions do not. Each of these is a search problem. None of them is a training problem.

I still want an oracle. But the experiment taught me where the memory actually lives: in the index, the files, and the discipline of writing things down. The weights kept nothing. The index kept everything. I know where I will spend my next evening.

Sources:
[1] LoRA as parametric memory: a failed experiment, Z. Peng (2026). zedongpeng.com
[2] AI memory benchmarks 2026, mem0 (2026). mem0.ai
[3] Introducing contextual retrieval, Anthropic (2024). anthropic.com/engineering
[4] llm-wiki pattern, A. Karpathy (2026). gist.github.com
[5] Continual learning via sparse memory finetuning, J. Lin et al., Meta (2025). arxiv.org
[6] Research notes: filesystem memory scores 74 percent on LoCoMo, Letta (2025). letta.com