BM25 vs Dense Retrieval and SPLADE for Production RAG

You have a working RAG prototype. The embeddings return reasonable results on your eval set, the LLM answers look fine, and now you need to commit to the retrieval layer you will actually maintain under real traffic. This is where most builders lose months. They pick the algorithm that won a leaderboard, wire up a vector index, and discover weeks later that the index does not fit in VRAM, that p95 latency has doubled, or that the model silently returns garbage on rare entity names.

The BM25 vs dense retrieval debate gets settled on accuracy far too early. On a single consumer GPU with 16 to 24 GB of VRAM, what actually decides which baseline ships is a constraint satisfaction problem over three things at once: the memory ceiling the index hits at build time, the p95 latency it hits at query time, and the specific failure mode it introduces when the query distribution drifts. BM25, dense, and SPLADE each survive one production scenario and quietly break another. The useful answer is a mapping from your scenario to a baseline, not a single winner.

Reframing BM25 vs dense retrieval as a constraint problem

Start with the arithmetic, and name the assumption that moves it most. A 7B model in fp16 takes about 14 GB of VRAM for weights alone on a 16 GB card. That is unquantized inference. Apply int8 quantization (bitsandbytes) and the same 7B drops to roughly 7 GB; int4 (GPTQ, AWQ) takes it under 4 GB. The memory floor is not fixed, it is a dial, and quantization turns it. At fp16, KV cache and activations at 4k to 8k context add 1 to 3 GB, so the real floor is 15 to 17 GB before any index exists. At int4, that floor falls to 5 to 7 GB, reopening several gigabytes for a dense index. A dense index for 2 million passages at 768 dimensions still needs roughly 6 GB of vectors plus graph overhead, so even at int4 both will not always fit. A BM25 index for the same corpus fits under 1 GB and runs on CPU regardless.

Three constraints decide which baseline ships, and each flows from that arithmetic.

Index-time memory. Whether your LLM floor is 5 GB (int4) or 15 GB (fp16), the gap between a sub-1 GB BM25 index and a 6 GB dense index determines whether you ship or thrash. Dense stores a fixed-length vector per passage plus a proximity graph. SPLADE stores an expanded sparse vector whose non-zero count grows with expansion behavior. BM25 stores only a lexicon and term statistics.

Query-time latency. On a single consumer GPU, p95 latency is governed by how much of the index fits in RAM or VRAM, not by raw model FLOPs. An index that spills to host memory pays a PCIe transfer tax on every query. This structurally favors BM25, whose sub-1 GB index almost never spills, and penalizes large SPLADE indices once the corpus scales.

Failure mode under shift. Every baseline has a characteristic way it breaks when queries look unlike the corpus or the training data. The right retrieval baseline is determined by the joint shape of corpus size, query distribution, domain stability, and hardware budget, not by a single benchmark number.

Before picking a winner, you need to know what each baseline actually stores, what it costs, and how it fails.

BM25, footprint and failure mode

The constraint verdict for BM25 is simple: its index almost never spills from RAM, its query cost is near zero, and its failure mode is the cheapest of the three to detect in production logs. BM25 is a probabilistic ranking function over an inverted index, combining term-frequency saturation with inverse document frequency and length normalization. The original BM25 paper frames it in a probabilistic relevance framework, and it remains the default scorer in Elasticsearch, whose BM25 variables explainer covers the tuning knobs.

Memory: a sub-1 GB index that never competes with the LLM

For under 100k passages, a BM25 index takes tens of megabytes. At 1 to 5 million passages, hundreds of megabytes to roughly 2 GB. BM25 stores only a lexicon and term statistics, not a vector per passage. No GPU required; one index serves one user or a hundred.

Latency: lexicon lookups, no encoder pass

Query cost is tokenization and arithmetic over postings lists. p95 stays flat as the corpus grows because the index fits in RAM, with no query-time model inference competing for VRAM.

Failure mode: exact-term match that dense collapses on

When queries are keyword heavy, when exact identifiers and rare terms matter (legal codes, product SKUs, error strings, chemical identifiers), BM25 outperforms dense because it does not smooth anything away. The symptom that BM25 is your baseline and not dense: keyword queries work, natural-language queries return nothing.

Where it loses: vocabulary mismatch. A paraphrased query with no term overlap returns nothing. The early symptom: users who type full questions get zero hits while keyword users succeed. We map that failure precisely in a later section.

Dense retrieval, semantic wins and silent collapses

What blocks dense retrieval on consumer hardware is the memory budget the index demands alongside the LLM, not the semantic quality. A 16 GB GPU already running a 7B or 8B model has little VRAM left, and dense competes for the same memory the LLM lives in.

Do the arithmetic. A 768-dim float32 vector is 3 KB per passage (768 times 4 bytes), so a million passages is roughly 3 GB of raw vectors before any index overhead. The DPR paper showed dense beating BM25 on open-domain QA, but that result presumes the index fits alongside the model. On a GPU where the LLM alone takes 15 to 17 GB, that vector footprint plus the proximity graph is the difference between shipping and thrashing. Malkov's HNSW paper describes the layered proximity structure that enables fast approximate search, but its multiple edges per vector inflate the footprint beyond raw embedding bytes, which is why FAISS memory discussions surface as a recurring operational complaint.

What dense buys per gigabyte of VRAM

Spend 3 to 6 GB on a dense index alongside a 7B model and you get three capabilities BM25 cannot match at any price. Semantic recall on paraphrase: "how do I reset my password" retrieves a document about credential recovery with zero keyword overlap. Zero-shot generalization within the training distribution: synonyms map to nearby vectors without labeled data, as long as queries stay in-domain. Natural-language query handling: users who type full questions get better results because the embedding captures intent, not surface form. Each is a query class BM25 returns nothing for, and together they justify the gigabytes spent.

Where dense breaks under VRAM pressure

The collapse arrives on exactly the queries that justify the cost. Dense degrades sharply on rare, out-of-domain, or entity-heavy queries because the embedding model has not seen those terms in distribution. The BEIR out-of-domain analysis documents this: dense retrievers fall off outside their training distribution while BM25 holds steady. On consumer hardware the collapse is doubly painful, because you paid that vector footprint and still get garbage on the queries that matter most. The unshippable scenario: a large corpus, an entity-heavy query distribution, and a single GPU where the dense index cannot fit beside the model.

SPLADE, learned sparse retrieval at a memory cost

SPLADE inherits the operational cost of both parents. The SPLADE paper introduced sparse lexical expansion, runnable via sentence-transformers SparseEncoder without a custom training loop. The representation is technically sparse, but operationally closer to dense than BM25.

Memory: learned expansion drives index bloat

A BM25 document has roughly 200 unique terms. SPLADE's expansion produces thousands of non-zero weighted terms per passage, each needing an inverted-index entry. The expanded index can be several times the size of a BM25 index at the same corpus, pushing memory into the range where it competes with the LLM for VRAM.

Latency: encoder pass competing with the LLM

Producing the sparse query vector requires a transformer forward pass, meaning GPU inference competing with the LLM for VRAM and compute. Unlike BM25's lexicon lookups, every query pays an encoder cost on the same GPU.

Failure mode: dense-level cost without dense-class guarantee

SPLADE targets dense-class recall on zero-shot benchmarks designed for out-of-distribution evaluation, like BEIR, while retaining some of BM25's exact-match behavior, though realized performance varies across tasks. The production risk: paying dense-level memory and latency without consistently achieving dense-level recall. SPLADE is a defensible single-index bet when your query distribution mixes keyword and semantic patterns and maintaining two indexes is not feasible.

Where each baseline breaks in production

Every baseline has one failure mode that dominates. Recognizing the early symptom is more valuable than chasing a recall point on a benchmark.

The latency failure deserves emphasis. The MS MARCO leaderboard ranks systems by retrieval quality, not by the memory pressure they create on your hardware. A system that scores well there can still be unshippable if its index spills out of VRAM and p95 spikes. This is why a BM25 vs dense retrieval comparison that only checks recall misses what actually takes your system down.

There is also a hidden constraint most retrieval comparisons ignore: index update cost.

• BM25. Insert a document into the inverted index; term statistics update cheaply.
• Dense. Any schema or chunking change forces re-embedding, because vectors are only valid against the chunking that produced them.
• SPLADE. The inverted index updates cheaply, but the encoder must run over changed passages to produce their sparse vectors.

If your corpus changes weekly, this is a recurring GPU-hours line item that can dwarf the original indexing cost.

Hybrid retrieval, when two indexes beat one

Hybrid is the most overprescribed answer in RAG retrieval. For a small corpus or a narrow, stable domain, one baseline already covers 90 percent of queries. Running a second index means two builds, two query paths, and two failure surfaces to monitor, all to rescue a long tail you may not have yet. If your domain is stable and your eval set is clean, ship a single baseline first and measure where it breaks before paying for hybrid.

The counterpoint is real. Hybrid BM25 plus dense with reciprocal rank fusion earns its cost in three situations: mixed query types where you cannot predict whether users type keywords or full questions, mid-size corpora where neither BM25 nor dense alone reaches acceptable recall at k, and shifting domains where the failure mode is uncertain. RRF is robust because it needs only ranks, not calibrated scores, so you never normalize two incomparable distributions. The practical pattern: ship BM25 only, watch for semantic misses in logs, then add dense as a second index once the misses justify the complexity.

For consumer hardware, hybrid is workable because the BM25 index is cheap enough that running it alongside dense rarely breaks the memory budget. You pay complexity, not hardware. The ColBERT paper offers a different architecture: late interaction over token-level embeddings that can outperform single-vector dense, at a storage cost that makes it a heavier lift than two-index hybrid on a single GPU.

A decision framework for consumer hardware

The BM25 vs dense retrieval comparison is now a routing decision: match each query type and corpus shape to the index that handles it. Use the table below as a starting point, then validate with a small eval set.

The two questions that most often flip the recommendation are domain stability and update frequency. A stable, in-domain corpus rewards dense. A shifting, out-of-domain, or frequently updated corpus pulls you back toward BM25 or hybrid, because dense's collapse and re-embedding cost show up exactly when the corpus moves.

Picking your retrieval baseline, a five minute checklist

Answer these five questions and you will land on a defensible baseline without running a full benchmark.

1. Corpus size. Under 100k passages, memory pressure is low and dense is viable. Over 1 million passages, index fit starts to dominate, and BM25 or a BM25-heavy hybrid becomes safer.
2. Query type. Mostly natural-language and semantic? Lean dense. Mostly keywords, identifiers, or exact terms? Lean BM25. Mixed? Plan for hybrid.
3. Domain stability. In-distribution and stable? Dense generalizes well. Out-of-domain, multilingual, or entity-heavy? BM25 or hybrid to hedge dense's collapse.
4. Hardware budget. How much VRAM and RAM is left after the LLM loads? Under a few gigabytes, BM25 is the safe floor. Five to ten free gigabytes opens up dense. SPLADE needs both room and an encoder pass at query time.
5. Update frequency. Static corpus? Any baseline works. Weekly or daily updates? BM25 updates cheaply, dense requires re-embedding changed chunks, and SPLADE needs encoder passes on changed passages.

If three or more answers point to BM25, ship BM25 first and add dense as a second index only if eval shows semantic misses. If three or more point to dense, ship dense but keep a BM25 index warm for the rare-term queries your eval set probably underrepresents. If you cannot predict your query distribution at all, ship hybrid from day one and accept the operational cost as the price of not guessing wrong.

The retrieval baseline worth shipping is the one whose failure mode you can survive when traffic arrives, whose index fits beside your LLM, and whose latency stays flat as the corpus grows. Leaderboard scores do not tell you any of that. Pick the baseline that breaks in a way you can detect and recover from, and you will ship a RAG system that lasts.