RAG in Production: Lessons from Building Document-Intelligence Pipelines

The tutorial version of RAG is: embed your documents, store the vectors, retrieve the top-k matches for a query, stuff them into the prompt, done. That version works on a demo dataset of twenty clean PDFs. It does not survive a real document set — inconsistent formats, scanned pages, tables, and documents that update over time.

What RAG actually solves

Retrieval-augmented generation exists because a model's training data is static and general, while your documents are specific and current. Instead of fine-tuning a model on your data — slow, expensive, and stale the moment your data changes — you retrieve the relevant pieces of your data at query time and give them to the model as context. It's a good default for document-heavy AI features, and a bad default when the real problem is something else (more on that below).

Where the naive pipeline breaks

Chunking. Splitting documents into fixed-size chunks ignores their structure. A table split across two chunks loses its meaning entirely; a chunk boundary that falls mid-sentence hurts retrieval quality in ways that are hard to debug because the failure looks like "the model just didn't find it," not "the chunking broke it."

Inconsistent document formats. Real document sets — invoices from different suppliers, contracts from different templates, scanned forms — don't share a layout. A pipeline that assumes one document template will silently produce garbage extraction on the other nine templates it wasn't tested against.

Retrieval precision, not just recall. It's easy to retrieve something relevant. It's harder to retrieve only what's relevant and rank it correctly — irrelevant chunks in the context window don't just waste tokens, they measurably degrade answer quality by diluting the model's attention.

Hallucination despite retrieval. Giving a model the right context doesn't guarantee it uses it faithfully. Models still sometimes blend retrieved content with prior knowledge, especially when the retrieved passages don't fully answer the question. Retrieval reduces hallucination; it doesn't eliminate it.

What actually helps

• Structure-aware chunking — split on document structure (headings, table boundaries, sections) rather than a fixed character count, and keep tables intact as single chunks with their headers preserved.
• Metadata filtering before vector search — if you know the document type, date range, or source, filter first and search within that subset. It's cheaper and more precise than relying on the embedding model to encode that distinction.
• Re-ranking — a fast first-pass retrieval followed by a more expensive re-ranking step on the top candidates consistently improves precision over a single-pass vector search.
• Grounding checks — for higher-stakes use cases, add a validation step that checks the model's answer actually traces back to the retrieved content, and surface "I don't have enough information" as a legitimate output rather than letting the model guess.

Evaluating quality, not vibes

"It looks right when I try it" is not an evaluation strategy. Build a small held-out set of real queries with known-correct answers, and track retrieval precision/recall and answer faithfulness separately — a pipeline can retrieve the right passage and still generate a wrong answer, or retrieve the wrong passage and still generate a right-sounding one. You need to know which failure mode you're looking at.

When fine-tuning beats RAG

If the problem isn't "the model doesn't know this fact" but "the model doesn't produce output in the style, format, or tone I need," RAG won't fix it — that's a fine-tuning problem, or a prompt-engineering one. Matching the technique to the actual failure mode, rather than defaulting to RAG because it's the familiar pattern, is most of what separates a document-intelligence pipeline that works from one that technically runs.