Reranking made my search slower and worse

I added local semantic search to a small knowledge base, and the default LLM reranker took 33 seconds to put the wrong document first. Every cheaper path got it right, which I only found out by asking questions I already knew the answers to.

I keep a knowledge base of Markdown notes about what I investigated, what I decided and what broke, and I wanted agents to search it before starting work instead of working everything out again. qmd does that locally. It runs BM25 keyword search and vector search over local embeddings, fuses the two with reciprocal rank fusion, and then by default runs an LLM reranking pass over the fused candidates.

It installed cleanly and returned results, and I could easily have stopped there.

Test with an answer you already know

What I did instead was ask it something I already knew the answer to: why an FTP mirror had failed its TLS check. There’s exactly one note about that.

The default query came back after 33 seconds with an unrelated design document as the top hit, at 38 %. The right note wasn’t in the visible results at all.

Once I ran the stages separately, the problem was obvious:

Every cheap path got it right, and only the expensive one failed. After that I ran five questions with known answers, all with reranking turned off. Four put the exact note first, and the fifth put it second, behind a note that really was related. Across all of this, reranking never improved a result, and it broke one.

Why it failed

The corpus was eleven documents. At that size, fusion already separates signal from noise. A reranker earns its cost by reordering a long tail of plausible candidates, and here there wasn’t one. So it spent 30 seconds shuffling a handful of documents and promoted the wrong one.

I put that down to the corpus being small, so I don’t read it as a verdict on reranking in general. I turned it off and wrote down why, with a note to re-test once there are around a hundred documents. If you disable a default without recording the reason, someone will eventually turn it back on to be helpful.

A visible GPU is not a usable GPU

The 33 seconds made me suspect the GPU wasn’t being used, and it wasn’t. On WSL2, seeing the card in nvidia-smi only tells you the driver stub is there. node-llama-cpp inspect gpu tells the two apart: “NVIDIA driver is installed, but CUDA runtime is not.” Without the CUDA toolkit inside the distribution, inference quietly runs on the CPU. The only symptom is a wall time several times longer than it should be, which shows up as a gap between wall-clock and user time.

I left it like that. CUDA would make reranking fast, but it wouldn’t make the results any better.

Two smaller things

My knowledge base has index files that copy each note’s summary into a table. When those got indexed, they competed with the notes they summarise for result slots, and they won because they’re shorter. So I excluded them from the index.

The other change was attaching context to collections and directories. A one-line description per topic travels with every matching chunk, so whoever reads a result knows what kind of document it came from without opening it. That took five minutes, using data the root index already had.

Getting results back didn’t tell me anything about whether they were right. The five questions with known answers took a few minutes, and they caught a default that would otherwise have stayed on.