The Weights Kept Nothing. So I Built a Measuring Machine

The last post ended with a list. I had trained a small language model on my entire knowledge base, watched it answer 0 of 131 questions correctly, and closed with three fixes that I claimed would help my setup more than any training run: an alias layer, chunk context, and a threshold that admits when the notes hold no answer. Three search problems. Not one of them a training problem.

Then I spent a week measuring all three. One shipped. Two died. And the week produced something that was not on the list at all, which turned out to be worth more than anything that was.

The machine came before the experiments

Before touching the search, I froze a test set. 178 questions, each written against a verified source and pointing at the file that answers it, plus 19 questions my notes cannot answer at all, which measure whether the system knows what it does not know. Then I locked the file and stopped editing it.

The loop around it is unglamorous. Every night the frozen questions run against a frozen copy of the search index, the run appends one row to a history file, and a flag goes up when a number drops. Nothing about it is clever. What it changes is the kind of sentence I am allowed to say about my own system. “This feels better” is no longer available. There is a row, and the row has a number in it.

The one that worked

The alias layer was the cheapest of the three and the only one that survived.

My knowledge base already held a knowledge graph with 94 entity aliases: nicknames mapped to people, abbreviations to concepts, project shorthand to project files. The search had never once consulted them. So I added a lookup lane. A query that matches an alias pulls the target file in as a high confidence candidate, and those hits are protected from being reranked back into obscurity.

The category the change targets went from 0.30 to 0.43 recall at five. Everything else moved by a point or two. And the mean reciprocal rank fell by two points, because forcing alias hits to the front pushes better ranked results down elsewhere. That cost is real and I kept it anyway, since the questions I actually ask are full of nicknames and abbreviations. The trade is visible, which is the entire reason for having the machine.

None of that was new knowledge. The graph had been sitting in the same repository as the search, 94 rows deep, for months. All I did was connect two things I already owned.

The two that died

Chunk context was the fix I felt most confident about. Anthropic’s recipe puts a short model-written description in front of every chunk before embedding it, and they measured 49 percent fewer retrieval failures with it [2]. I built it on a copy of the index, re-embedded everything, and ran the test set against it.

Exactly zero change. Not small, not within noise. All six metrics identical, all 178 top scores identical, and I verified byte for byte that the run really had used the enriched copy, because a null result that clean is usually a plumbing bug. An independent replication explains the conditions: the effect concentrates at low k and depends on chunk size [3]. My chunks are large and my cross-encoder decides the ranking, so the embedding vector barely gets a vote. The recipe is real. It has nothing to grip in my setup.

The threshold died differently. The idea was a score below which the system says “no confident answer” instead of handing over its best five guesses. I measured the separation between answerable and unanswerable questions and got an area under the curve of 0.74, which sounds respectable and is useless in practice. There is no cutoff. At every setting, each hallucination I catch costs three real answers. Meanwhile the models handle this fine without a threshold: even the failed fine-tune declined 10 of its 19 unanswerable questions correctly. The practitioner literature calls global score thresholds folk defaults and points at a reranker with an explicit null class instead [1].

The rule I could not measure my way around

One more thing I tested, out of curiosity rather than off the list: the rule of thumb that says if your knowledge fits in 200k tokens, skip the retrieval infrastructure and put everything in the prompt [6]. The source files behind my test set need 418k. Given a 117k snapshot of them, the model answered 4 of 5 questions whose sources made the cut, correctly declined the ones it could not see, and hallucinated once with the answer sitting right there in context.

The rule holds. My corpus is above the line. Retrieval stays.

What was not on the list

The thing worth more than any of the three fixes is the loop itself, and what became possible once measuring was routine.

Every night an agent now reads what changed in my knowledge base since its last run, the actual git diff, and distills it into curated insight files, clearly marked as unreviewed. The labs call this sleep-time compute [4] or dreaming [5]: the system thinks during downtime and turns raw episodes into something durable. I would not have trusted myself to build that in March. I trust it now, because if it quietly makes the search worse, a row in a history file will tell me by morning.

Negative results are infrastructure

In a system you measure nightly, a negative result is not a dead end, it is calibration. I know what training cannot do here, what a score threshold cannot do, what chunk context cannot do in this setup, and where the 200k rule stops applying. The list of things that do not work is the map of the terrain. Without it I would still be shipping changes because they felt right.

The weights kept nothing. The measuring machine keeps everything.

Sources:
[1] The retrieval emptiness problem, T. Pan (2026). tianpan.co
[2] Introducing contextual retrieval, Anthropic (2024). anthropic.com/engineering
[3] Lessons from testing contextual retrieval on multiple datasets, Datapizza (2026). datapizza.tech
[4] Sleep-time compute: beyond inference scaling at test-time, Letta and UC Berkeley (2025). arxiv.org
[5] Dreaming: better memory for ChatGPT, OpenAI (2026). openai.com
[6] RAG vs fine-tuning in 2026, U. Malik (2026). umesh-malik.com