10 RAG Pipeline Mistakes That Reduce Production Quality
The most common RAG pipeline mistakes teams make in production — from bad chunking and missing reranking to invisible ingestion failures and no evaluation harness.
A RAG answer depends on the documents retrieved, their freshness, and how the model uses them. Evaluate ingestion, retrieval, and generation separately so a failure in one stage does not send you tuning the wrong component.
1. Blaming the LLM When Retrieval Is the Problem
When an AI assistant returns a wrong or irrelevant answer, the instinct is to tweak the prompt or switch to a larger model. One possible cause is upstream: the retriever did not surface the right document.
Before changing anything else, check what chunks the retriever actually returned for a failing query. Add logging that records retrieved chunk IDs and scores alongside every response. Compare the retrieved evidence with the generated answer to locate the failure.
2. Skipping the Evaluation Harness
Teams that build RAG without an evaluation set cannot tell if a change made things better or worse. They make changes based on anecdote. They ship regressions without realizing it.
Build a labeled query set before you tune anything. Even 50 manually reviewed query-answer pairs lets you compute Recall@5 and catch regressions. This is not optional infrastructure — it is how you know whether the system works.
3. Using Fixed-Size Chunking Without Checking Document Structure
Fixed-size chunking is a fine default, but not if your documents contain tables, PDFs with multi-column layouts, or structured data like JSON. Raw PDF extraction followed by fixed-size splitting produces chunks that contain half a table header, some boilerplate footer text, and a fragment of actual content. The embedding of that chunk is noise.
Pre-process documents before chunking. Extract structure, remove headers and footers, handle tables separately, and validate chunk content before indexing.
4. Not Handling Document Updates
Documents change after indexing. Outdated chunks produce confident wrong answers, which is worse than no answer.
Track a hash or last-modified timestamp for every source document. On update, delete all associated chunks from the vector index before inserting new ones. Run a periodic reconciliation job that compares source state to index state and queues stale chunks for re-ingestion.
5. Missing Metadata on Chunks
Chunks without metadata are hard to filter, impossible to audit, and difficult to remove when the source document changes. Every chunk should carry at minimum: source document ID, chunk index within the document, section or page, and ingestion timestamp.
Metadata is also how you implement multi-tenant isolation — ensuring that queries from one user or organization never retrieve content from another. This is not a nice-to-have; it is a security requirement for any multi-tenant deployment.
6. Ignoring Retrieval Noise
Retrieval noise occurs when irrelevant chunks make it into the context window. The LLM sees ten chunks, two of which are relevant. The other eight pull the generation toward irrelevance or contradiction.
Noise is a retrieval precision problem. Solutions include: tighter similarity score thresholds, metadata pre-filtering, and reranking. A cross-encoder reranker can reduce noise in the initial retrieval set; evaluate its accuracy and latency on representative queries.
7. No Reranking Step
Initial ANN retrieval optimizes for recall — it finds candidates that might be relevant. Reranking optimizes for precision — it identifies which candidates are actually relevant. Running retrieval without reranking is like running a search engine that returns 100 results and stops, trusting the user to find the right one.
Add a cross-encoder reranker between retrieval and generation. Retrieve 20-50 candidates, rerank, pass the top 5 to the LLM. Measure the effect on answer quality and latency before adopting it.
8. Ignoring Ingestion Failures
When document ingestion fails silently, the LLM still returns answers — just not ones grounded in the missing documents. Unlike retrieval failures, which produce wrong answers the user can sometimes recognize, ingestion failures are invisible: the knowledge simply does not exist in the index.
Log every ingestion job. Track success and failure rates per document. Alert on ingestion failure rates above a threshold. Implement a dead-letter queue for failed documents so they can be retried without re-running the full pipeline.
9. Using One Embedding Model for Everything
The embedding model you use matters for retrieval quality, and different models have meaningfully different strengths. A general-purpose embedding model trained on web text may underperform a domain-adapted model on legal, medical, or financial documents. Multilingual corpora need multilingual embedding models.
Evaluate multiple embedding models against your labeled query set before committing. The right model is determined by your corpus and query distribution, not by benchmark leaderboard position.
10. Treating RAG as a One-Time Build
RAG systems degrade over time. New documents are added in formats the chunker was not designed for. User query distributions shift. Embedding models are updated and older vectors become stale relative to new model outputs.
Treat your RAG pipeline as a running system, not a completed project. Schedule periodic evaluations against your labeled set. Monitor retrieval latency, chunk count per query, and answer quality feedback. Set up alerts for evaluation metric drops. Use those alerts to trigger an investigation into the affected stage.