Where vector search misses
Embeddings find meaning and lose precision. BM25 finds exact strings and loses meaning. Production retrieval runs both.
Embeddings are good at meaning and bad at precision. Ask for “contract termination notice periods” and vector search will surface passages about ending agreements even when no word matches. Ask for part number “KTM-4402-B” and it can shrug, because a code that appears nowhere in training data carries almost no semantic signal. A keyword index has the opposite temperament. BM25 has been ranking exact terms since long before anyone said retrieval-augmented generation, and it still wins whenever the query contains names, codes, or jargon that must match literally.
The fix is to stop choosing. Candidates come from several indexes at once: BM25 for exact terms, dense vectors for paraphrase, and a graph when the question hops between entities. A reranker then reads the merged set against the query and keeps only what survives. The generator sees five good passages instead of forty mediocre ones.
None of the weights come from a template. Chunk sizes depend on whether your documents are contracts or chat logs, the embedding model depends on language and domain, and the blend between indexes depends on how your people ask, which you only learn by measuring recall on real questions.
None of this is exotic. It’s the lesson search teams learned twenty years ago, relearned with vectors. The part that stays hard is chunking tables and diagrams, and that deserves its own note.