I built hybrid retrieval, then measured it. It tied.

I built a RAG application called DocuMind. You upload a PDF, ask questions about it, and get answers drawn only from that document, with citations back to the exact source chunk. It is live at documind-web-mu.vercel.app.

Retrieval is hybrid. Two searches run over the same Postgres table and their results are fused into one ranking. I built it that way because every article I read said hybrid beats plain vector search.

Then I tested whether that was true for my system. It was not.

How the retrieval works

Every chunk of an uploaded document is stored in one Postgres table with two things attached: an embedding, and a tsvector of its text.

Semantic search runs over the embeddings using pgvector, with cosine distance and an HNSW index. This finds chunks that mean the same thing as the question, even when they share no words with it. Ask “what happens if a family member dies” and it returns the bereavement leave policy without either of you using the word bereavement.

Keyword search runs over the tsvector using Postgres full text search and ts_rank. This finds chunks that contain the actual words. It is exact where the embedding is fuzzy.

Each search returns its own ranked list. Reciprocal Rank Fusion merges them. Every chunk scores 1 / (60 + rank) for each list it appears in, and those scores are summed. A chunk that ranks fourth in both lists beats a chunk that ranks second in one list and is absent from the other. The constant 60 is the standard damping value from the original RRF paper. It stops the top position from dominating everything below it.

That is the whole mechanism. Two opinions about relevance, combined so that agreement counts for more than any single strong opinion.

The eval

I wanted to know whether the keyword half was earning its place, so I built a test set.

Twenty-seven questions across four academic papers, about 302 chunks total. For each question I recorded in advance which chunk actually contained the answer. That gold label is the entire point. Without it you are reading outputs and deciding they look reasonable, which is not measurement.

Then I ran the same 27 questions three ways and measured top-1 recall, meaning how often the correct chunk came back in first position.

Vector search got every single question right on its own. Hybrid also got every question right. Adding keyword search to a system that was already perfect could not improve it.

Hybrid tied. It did not win.

The test set was the problem, not the result

The honest reading is not “hybrid retrieval doesn’t work.” It is “my test set could not tell these two apart.”

Four academic papers are semantically uniform prose. Every chunk is full sentences on a related topic, written in a consistent register. That is the best case for embeddings, and the worst case for showing where they fail. My questions were also natural language questions, which is exactly what dense retrieval is built for.

A test set where one method scores 100% has no room left to measure anything. The ceiling hid the comparison.

Keyword search scoring only 48% is the more interesting number, and it points at the same problem from the other side. On this corpus, exact word matching was actively bad, because the questions rarely reused the document’s phrasing. That is a property of my corpus, not a property of keyword search.

Why I kept hybrid anyway

I kept it, and I wrote down that I was keeping it as a bet rather than a proven improvement.

The reasoning is about the queries my eval never contained. Embeddings blur exact tokens. A policy code like RW-114, an error string, a function name, a part number, an acronym that appears twice in a 300 page document: these are cases where the semantically nearest chunk is not the right chunk, and where a lexical match is not merely helpful but necessary. My four academic papers had almost none of these, so my eval could not see the gap.

That is a defensible reason to keep a component. It is not a reason to claim it improved anything, and the difference between those two statements is the only thing this whole exercise was about.

What a harder test set looks like

The next version needs questions the current one cannot ask:

• Exact identifiers. Policy codes, ticket numbers, part numbers, version strings. Cases where being semantically close is being wrong.
• A mixed corpus. Not four papers in one field. Documents in different registers, so that nearest in embedding space stops being a reliable proxy for correct.
• Distractor chunks. Passages that discuss the same topic but do not contain the answer, so that a near miss is punished rather than rewarded.
• Rare terms. Words appearing once or twice, where the embedding has thin signal and lexical matching has strong signal.
• Recall at 3 and 5, not just top-1. A ceiling at 100% on one metric hides everything. More metrics, more room to see a difference.

I would also build the eval before building the feature next time. I built hybrid retrieval, then measured it, and the measurement told me I had not needed to. Building the harness first would have shaped what I built rather than grading it afterwards.

The part worth keeping

The result I got is less useful than the habit. I had a thing I believed, I checked it, and it was not true, and the correct response was to write that down rather than round it up.

An unmeasured system does not have a performance. It has a story about its performance. Those are different, and only one of them survives contact with a real corpus.

Code is public at github.com/AditiV05.