Production RAG

The model only sees what you retrieved

Most disappointing RAG systems fail the same way: the passage that would have answered the question never made it into the prompt. Prompt engineering does not recover from that, and neither does a bigger model. The model is answering honestly from a context that does not contain the answer.

Retrieval sets a ceiling. Generation runs underneath it. That is why upgrading the model is usually the wrong first move.

Retrieval failure vs generation failure

From outside, both look like a wrong answer. Inside they are different bugs with different fixes:

Log the retrieved IDs

You cannot do this triage after the fact unless you stored what came back. Log, per request: the query, the retrieved chunk IDs, their scores, and which ones you actually put in the prompt after truncation.

log.info("retrieval", extra={
    "query": query,
    "candidates": [(c["id"], round(score, 4)) for c, score in scored],
    "used": [c["id"] for c in used],
})

That last field matters more than it looks. Systems commonly retrieve 10 chunks and then drop half of them to fit a token budget, so the chunk that would have answered the question was retrieved and discarded — which reads as a retrieval failure in every dashboard and is actually a budgeting bug.

The triage

For twenty real failures, ask whether the correct passage was in the retrieved set. If it was not, sections two through six apply and nothing about your prompt matters yet. If it was, go to citations and grounding.

What this pack assumes

You have embeddings going into a store and coming back out. If that is new, start with Embeddings & Vector Search. Examples here use Python and Postgres with pgvector to match it. The techniques are not Postgres-specific, though a few of the sharper edges are.

Related Concepts

External Resources

Score retrieval separately

End-to-end answer quality mixes two systems, so it cannot tell you which to fix. Score retrieval alone and you get a number you can act on.

You need questions paired with the chunk IDs that actually answer them. Thirty is enough to start. Build it from real questions users asked, not invented ones.

Recall@k

Of the k chunks you put in the prompt, did a correct one make it in?

This is a hard ceiling. Retrieve 5 chunks with recall@5 of 60%, and 40% of questions are unanswerable regardless of which model reads them. No prompt change moves that.

Gate on recall@k. Watch MRR second — a correct chunk at rank 9 of 10 competes with eight distractors for attention, and long-context attention is not uniform.

InformationRetrievalEvaluator

sentence-transformers computes all four against a labeled set:

from sentence_transformers.evaluation import InformationRetrievalEvaluator

evaluator = InformationRetrievalEvaluator(
    queries=queries,              # qid -> question text
    corpus=corpus,                # cid -> chunk text
    relevant_docs=relevant_docs,  # qid -> set of correct cids
    name="support-docs",
)
results = evaluator(model)

Note what this measures: the retriever in isolation, against the full corpus. It will not catch a bug in your WHERE clause or your token budget, because it never runs your query path. Keep a second, smaller set that hits the real endpoint.

Running it in CI

These are eval cases like any other — see Evals & Benchmarking for the harness. The difference is what you assert on: chunk IDs, not answer text. Assert on IDs and the test survives a reworded document; assert on answer text and it does not.

Related Concepts

External Resources

Where fixed-size splitting breaks

Split every 500 characters and you cut tables in half, separate headings from the paragraph they introduce, and strand pronouns from their referents. Each produces a chunk that can no longer answer a question it physically contains.

The splitter decides what is retrievable at all, which is why it is worth more attention than it usually gets.

Splitting on structure

Documents carry boundaries their authors intended. Use those first and fall back to length only inside a section that is genuinely too long.

• Markdown and HTML — split on heading level, keep the heading text with its body
• Code — split on function or class, never mid-body
• Tables — repeat the header row in every chunk of a long table
• Transcripts — split on speaker turn or topic shift, not character count

Choosing a chunk size

The dilution is the part people underestimate. One chunk gets one vector, so a 1,500-token chunk spanning four topics produces a vector that sits near none of them and gets retrieved for none of them cleanly. That is the mechanism behind "we made chunks bigger and recall got worse."

Overlap of 10-20% hedges against splitting mid-answer. It does not fix bad boundaries.

Metadata to store

chunk = {
    "text": body,
    "doc_id": doc.id,
    "heading_path": "Billing > Refunds > Timing",
    "source_url": doc.url,
    "updated_at": doc.updated_at,
    "model": "all-mpnet-base-v2",
}

doc_id is what makes deletes possible later. model is what stops you silently mixing vectors from two embedding models in one table — see index maintenance. heading_path gives you both a retrieval filter and a citation breadcrumb.

Then measure

Every claim above is a hypothesis about your corpus, not a rule. Change one thing, re-run recall@k, keep it only if the number moved.

Related Concepts

External Resources

The orphaned chunk

"Revenue grew 3% over the previous quarter" is close to useless alone. Which company, which quarter? Embedded on its own it lands in a generic neighborhood of revenue sentences and gets retrieved for the wrong questions.

Prepending context

Anthropic's Contextual Retrieval generates a short situating sentence per chunk and prepends it before indexing:

This chunk is from an SEC filing on ACME corp's performance in Q2 2023;
the previous quarter's revenue was $314 million.

Revenue grew 3% over the previous quarter.

What you show the user is unchanged. What changed is what you embedded and indexed.

Measured results

Against their retrieval benchmark:

The last two rows are the next two sections, and they stack.

Cost

Generating 50-100 tokens per chunk sounds expensive until you notice the document is identical across every one of those calls. Prompt caching is what makes it affordable: you cache the document once and pay full rate only for the context sentence each chunk generates.

It runs at index time, so it adds nothing to request latency. Budget it as a batch job that reruns when a document changes, which means it rides along with incremental re-indexing rather than being a separate pipeline. Price it against your provider's current cached-input rate before committing to it on a large corpus — that rate is the whole reason this is viable.

When it does not help

Chunks that are already self-contained — FAQ entries, product records, tickets with their own titles — gain little from a situating sentence. This pays off on long narrative documents where any given paragraph assumes everything before it.

Related Concepts

External Resources

Where vector search fails

Embeddings find things that mean the same. They are unreliable at finding things that are the same — an error code, a SKU, a surname, a config key. Search ERR_CONN_4021 and a dense retriever returns passages about connection errors generally, none containing that string.

Lexical search has the inverse profile. Running both and merging fixes an entire class of failure.

The vector side

SELECT id, text
FROM chunks
ORDER BY embedding <=> $1
LIMIT 50;

The lexical side

SELECT id, text, ts_rank_cd(tsv, query) AS rank
FROM chunks, websearch_to_tsquery('english', $1) query
WHERE tsv @@ query
ORDER BY rank DESC
LIMIT 50;

Use websearch_to_tsquery rather than to_tsquery for anything user-typed. It accepts quoted phrases and -exclusions, and it does not raise on stray punctuation — to_tsquery will happily error on a question mark from a search box.

Postgres ranking is not BM25

Worth being precise here, because it is widely misstated. ts_rank_cd is a cover-density measure, and the PostgreSQL documentation is explicit about the limit:

the ranking functions do not use any global information

No corpus-wide statistics, which means no IDF. A term appearing in every document is weighted the same as a rare one. That is exactly the signal BM25 exists to provide.

In practice this matters less than it sounds, because RRF only consumes rank order and the reranker rescores everything anyway. But do not describe it as BM25 in your own docs, and if true BM25 scoring matters for your corpus, that needs an extension or a separate search engine — it is not something you tune your way into with ts_rank_cd weights.

Merging with RRF

The two result sets have incomparable scores: cosine distance and ts_rank_cd live on unrelated scales. RRF ignores scores and uses only rank position.

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

k defaults to 60 and controls how much influence lower-ranked results retain — raising it flattens the curve so deeper results matter more. Nothing here needs fitting, and nothing needs refitting when you change embedding models.

Related Concepts

External Resources

Bi-encoders and cross-encoders

Your retriever is a bi-encoder: query and chunks are embedded separately, so chunk vectors are computed once and reused. That independence is what makes search over millions of chunks fast, and it is also the limitation — the chunk was embedded without ever seeing the query.

A cross-encoder takes query and chunk together and returns one relevance score. More accurate, far too slow to run over a corpus.

The two-stage pipeline

Retrieve 50-100 candidates cheaply, score just those, keep the top 5:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
scores = reranker.predict([(query, c["text"]) for c in candidates])
top = [c for _, c in sorted(zip(scores, candidates), reverse=True)][:5]

Choosing a model

L6-v2 is the usual pick. L12 costs roughly double the time for one hundredth of a point. At 100 candidates per query, L6 adds well under a tenth of a second on GPU — on CPU, measure it before you promise anyone a latency budget, because those throughput figures are not CPU figures.

Hosted reranking APIs exist and save you from serving a model, at the cost of another network round trip inside your request path.

What reranking cannot fix

Reranking moves MRR a lot and recall barely at all. It reorders what it was given; it cannot retrieve something the first stage missed.

So if recall@50 is already poor, a reranker will not save you and you should go back to chunking and hybrid search. The corollary is to retrieve wider than feels necessary — the reranker is what makes a wide, noisy candidate set safe to ask for.

Related Concepts

External Resources

Users do not type search queries

Real questions arrive as "does that work on the old plan?" — pronouns, missing subject, none of the vocabulary the documents use. Embedding that verbatim searches for the wrong thing.

Three techniques, in the order worth trying.

Resolving follow-ups

Cheapest, and the one most often skipped. Resolve the question against conversation history before retrieving:

rewrite = f'''Rewrite the follow-up question as a standalone search query.

Conversation:
{history}

Follow-up: {question}
Standalone query:'''

"does that work on the old plan?" becomes "does CSV export work on the Legacy Starter plan", which has something to match against.

Watch the failure mode: on a first turn with no history, a rewriter will sometimes invent specificity that was not there. Skip the rewrite when history is empty rather than trusting it to no-op.

Multi-query

Generate two or three phrasings, retrieve for each, merge with the same RRF from the previous section, then rerank the merged set. One small model call, and it covers vocabulary you did not anticipate.

HyDE

Questions and answers are written differently, which weakens direct question-to-chunk similarity. HyDE has a model write a hypothetical answer and embeds that instead of the question.

The generated answer may contain invented specifics, and that turns out not to matter — the encoder's compression discards false details while keeping topic and vocabulary, landing you near real documents. It was introduced for zero-shot retrieval with no labeled data, which is where most teams start.

The cost is a generation call before every search, on the latency path. Worth it when questions are short and documents are dense prose; usually not when your corpus is already question-shaped.

Measure each separately

All three add latency and two add cost. Add them one at a time against the labeled set, and keep only what moves recall@k.

Related Concepts

External Resources

Asking for citations

Citations are usually treated as a UI nicety. They are more useful as the mechanism that makes a wrong answer detectable — by a reviewer, by an automated check, and by the user.

Ask structurally, so they can be verified in code:

prompt = f'''Answer using only the sources below. Every sentence must cite
the source ID it came from, as [S1]. If the sources do not contain the
answer, say so and cite nothing.

{format_sources(chunks)}

Question: {question}'''

Assign short stable IDs at prompt-assembly time and keep the map back to doc_id and source_url.

Verifying them

An unchecked citation is decoration. Two assertions catch most of it:

• Every cited ID exists in the set you supplied. A [S7] when you passed five sources is fabricated, and that is a one-line check.
• Every sentence carries a citation, or the answer is an explicit refusal.

Both are code, not judgment, and both belong in the eval suite.

Detecting ignored context

The generation failure from the first section needs a grader that reads the answer and the sources together. Use a binary rubric, one claim at a time:

Does every factual claim in the answer appear in the cited source?
Answer yes or no.

Binary per-claim questions score far more consistently than "rate faithfulness 1 to 5." Grade with a different model than the one that wrote the answer.

Allowing "not in the sources"

A system that must always answer will always invent something. Give the model an explicit refusal option and put it in the eval set: a case whose correct answer is "the sources do not cover this" is one of the most valuable rows you have, because that is precisely where a confident fabrication does the most damage.

Related Concepts

External Resources

Incremental re-indexing

Full rebuilds are simple and get skipped because they are slow. Hash the chunk text and only re-embed what changed:

digest = hashlib.sha256(chunk_text.encode()).hexdigest()
if digest != stored_digest:
    embed_and_upsert(chunk_id, chunk_text, digest)

Deleting stale chunks

Upserts get built on day one; deletes get forgotten. A deleted source document leaves its chunks in the index indefinitely, which is how a system ends up citing a policy that no longer exists.

Two rules avoid it. Every chunk carries doc_id, so removing a document is one delete by that key. And re-indexing a changed document deletes its old chunks first — chunk boundaries move when text changes, so matching by chunk ID strands orphans.

Changing the embedding model

Vectors from two models are not comparable even at identical dimensionality, so swapping models means re-embedding everything. There is no partial migration.

The vector(n) column also pins the dimension, so a model with a different output size is a schema change, not a config change. Store the model name per row and treat a mismatch as a hard error rather than discovering it through mysteriously bad results.

The 2,000-dimension index limit

A vector column accepts up to 16,000 dimensions. HNSW and IVFFlat indexes only support up to 2,000.

That gap is a live trap: a 3,072-dimension embedding inserts fine, queries fine, and cannot be indexed. You get correct results by sequential scan and discover the problem as a latency cliff in production. halfvec raises the ceiling to 4,000, and binary quantization further, at a recall cost.

Filtering collapses top-k

The one most likely to bite. With an approximate index, filtering is applied after the index is scanned:

SELECT id FROM chunks
WHERE tenant_id = $2          -- applied after the scan
ORDER BY embedding <=> $1
LIMIT 10;

Ask for 10, and if tenant_id matches a small slice of the table you may get 2 — the scan found its candidates first and the filter deleted most of them. It is not an error and nothing logs a warning.

Fixes: enable iterative index scans so the scan continues until enough rows survive the filter, or use partial indexes per tenant.

Build and search settings

hnsw.ef_search defaults to 40 and is the recall dial at query time:

SET hnsw.ef_search = 100;

At build time, m (16) and ef_construction (64) trade index size and build time for accuracy, and the build is dramatically faster when the graph fits in maintenance_work_mem — Postgres emits a notice when it does not. Use CREATE INDEX CONCURRENTLY so the build does not block writes, and load your data before building rather than after.

Before rewriting your chunker, compare against an exact scan on a small sample. Chunks that appear missing are often indexed and simply not being reached.

Related Concepts

External Resources

Start from the symptom

The order that wastes the least time

Most teams work this backwards — prompt first, model second, retrieval last.

• Step 1 — label 30 questions with their correct chunk IDs.
• Step 2 — measure recall@k. This is your ceiling and it tells you whether anything else matters.
• Step 3 — if recall is low, fix chunking, then add hybrid search. These move it most.
• Step 4 — if recall is fine and answers still miss, add reranking to raise MRR.
• Step 5 — only now touch the prompt, and add citations so failures become visible.

The cheapest three

Each is roughly an afternoon and independently measurable:

• Hybrid search. One extra query, and it fixes an entire class of failure embeddings cannot address.
• Reranking. One model, 50 candidates down to 5, large jump in how often the best chunk lands first.
• Contextual chunks. A batch job at index time, reported at 35% failure-rate reduction alone and 49% combined with lexical search.

None needs a framework, a migration, or a bigger model. All three are measurable against the labeled set from section two, which is the only reason to believe they worked.

Related Concepts

External Resources