Franck PachotDatabase Developer Advocate
All minibooks
Database field guide · 35
Lexical, semantic, relational, and spatial evidence
Design hybrid retrieval as a multi-channel candidate and ranking system, distinct from any single vector or BM25 index.
Build the mental model before choosing the mechanism.
This minibook was AI-generated from Franck Pachot's archived blog posts. Links to the original articles are included for source context and verification.01
Lexical retrieval rewards explicit terms while vector retrieval captures semantic proximity. Neither is a fallback implementation of the other; hybrid systems retain both signals for candidate generation and ranking.
02
Fusion cannot recover a relevant document absent from every candidate list. Candidate depth, analyzer behavior, ANN recall, filters, and per-channel limits determine the pool before ranking begins.
03
BM25-like scores, cosine distance, spatial distance, and business priors have unrelated scales. Reciprocal-rank fusion combines order rather than pretending raw values share units.
04
Tenant, authorization, time, inventory, and geography may be eligibility rules rather than relevance hints. Enforce mandatory constraints in every retrieval path before returning results.
05
A more expensive model can rerank a small fused candidate set, but it cannot repair omissions upstream. Latency, token cost, and failure behavior require an explicit budget.
06
A single top-k metric hides why hybrid retrieval improved or regressed. Evaluate lexical-only, vector-only, fused, filtered, and reranked stages across navigational, semantic, rare-term, and local queries.
07
Concrete mechanics, diagnostic evidence, and executable patterns to carry into a real system.
Run lexical and semantic retrieval independently and preserve ranks before fusion. This makes omissions and channel-specific regressions diagnosable.
lexical = top_n_bm25(query, 100)
semantic = top_n_vector(embed(query), 100)
# Persist document ID, rank, raw score, index version, and filter.RRF avoids assuming that lexical scores and vector distances have the same scale. Missing candidates contribute zero from that channel.
rrf(id) = sum(1 / (60 + rank[channel][id])
for channel in channels if id in rank[channel])Authorization and tenant predicates must constrain every candidate channel. A final filter is defense in depth, not permission to retrieve across boundaries carelessly.
eligible_lexical = lexical_search(query, tenant_id, acl)
eligible_vector = vector_search(embedding, tenant_id, acl)
assert all(can_read(user, row) for row in fuse(...))Compare channel, fusion, and reranking quality using the same judgments. Report latency and candidate counts so a quality gain has an explicit cost.
stages = ['lexical', 'vector', 'fused', 'reranked']
for stage in stages:
report(nDCG_at_10, recall_at_100, p95_ms, candidates)08
Optional deep dives with the complete experiments and product-version context behind this guide.