How Production RAG Systems Actually Work

A retrieval-augmented generation demo takes about an afternoon: chunk some documents, embed them, search, stuff the results into a prompt. It works impressively well on the documents you tested with.

The gap between that and something people rely on daily is not model quality. It is seven unglamorous components around the model, each of which fails in a way the demo never showed you.

#The shape of the thing

Two paths, not one. They have completely different performance characteristics, and conflating them is the most common structural mistake.

Ingest is batched, restartable and can take minutes. Query is interactive and has a latency budget measured in hundreds of milliseconds. Scaling them together wastes money on one and starves the other.

#1. Extraction is where most pipelines quietly break

A two-column PDF read naively produces interleaved nonsense: the first line of column one, then the first line of column two, and so on. That text gets chunked, embedded, and stored. Nothing errors. Retrieval quality is simply worse than it should be, forever, and no amount of prompt engineering recovers it.

The fix is a per-format extractor chain with a confidence check, and a rule that low-confidence extraction is flagged rather than indexed:

def extract(path: Path) -> Extraction:
    for extractor in EXTRACTORS_FOR[path.suffix]:
        result = extractor(path)
        if result.confidence >= MIN_CONFIDENCE:
            return result

    # Indexing text we cannot read is worse than not indexing it: the document
    # becomes permanently unfindable *and* pollutes neighbouring results.
    return Extraction(text="", confidence=0.0, status=Status.NEEDS_REVIEW)

An unreadable document that reports itself as unreadable is a support ticket. An unreadable document that silently indexes garbage is a product that "just isn't very accurate".

#2. Chunking on structure, not on character count

The default advice — 512 tokens with 64 overlap — is a reasonable fallback and a poor default. Fixed-size splitting routinely severs a definition from the term it defines, which makes that passage unretrievable by the exact question that needs it.

FIXED-SIZE SPLIT                    STRUCTURE-FIRST SPLIT

┌─────────────────────────┐         ┌─────────────────────────┐
│ ...renewal terms shall  │         │ 4.2 Termination         │
│ apply. 4.2 Termination  │         │ Either party may        │
│ Either party may        │         │ terminate with 30 days  │
└─────────────────────────┘         │ written notice...       │
┌─────────────────────────┐         └─────────────────────────┘
│ terminate with 30 days  │         ┌─────────────────────────┐
│ written notice...       │         │ 4.3 Renewal             │
└─────────────────────────┘         │ ...                     │
                                    └─────────────────────────┘
"How do I terminate?" matches        The heading travels with its
neither chunk cleanly.               clause. Retrieval works.

Split on the document's own structure first — headings, clauses, list items, paragraphs — and only then enforce a token ceiling on anything still too large. Carry the heading path into the chunk text so the passage remains self-describing once it is torn out of its document.

#3. Embedding is a cache problem

Embedding is the dominant cost of ingest, and the naive implementation re-embeds everything on every re-ingest.

Key chunks by a hash of their content. Re-ingesting a document that changed one paragraph then becomes a set difference: embed what is new, delete what is gone, keep what is unchanged.

chunk_ids = {sha256(c.text) for c in chunks}
existing  = store.ids_for_document(doc.id)

to_add    = chunk_ids - existing
to_delete = existing - chunk_ids
# Everything in the intersection is already correct. Don't touch it.

vectors = embed_batched([c for c in chunks if sha256(c.text) in to_add])
store.upsert(vectors)
store.delete(to_delete)

Two things fall out of this for free: retries resume rather than restart, and a document re-uploaded unchanged costs nothing.

#4. Hybrid retrieval, because both methods have blind spots

Dense vector search understands paraphrase and misses exact identifiers. Lexical search — BM25 — nails identifiers and misses paraphrase entirely.

Run both, fuse the ranked lists, then rerank the fused window. This is why production search is almost never purely one or the other.

#5. Reranking is where the accuracy actually comes from

First-pass approximate nearest-neighbour ordering is frequently wrong in the top five — which is exactly the range that reaches the model. A cross-encoder reads the query and the passage together rather than comparing two precomputed vectors, and is substantially more accurate at ordering.

It is also far too slow to run over a whole corpus. So: retrieve wide, rerank narrow. Top-50 from hybrid retrieval, reranked down to the five that enter the prompt. The cost is bounded because the window is fixed.

#6. Grounding is structural, not a polite request

"Please cite your sources" in a system prompt produces citations that are sometimes real.

Instead, give each retrieved chunk a stable identifier, require citations in the response schema, and drop any sentence whose citation does not resolve before it reaches the UI:

answer = model.generate(prompt, response_schema=CitedAnswer)

valid = {c.id for c in retrieved}
answer.claims = [c for c in answer.claims if c.citation in valid]

if not answer.claims:
    return Abstention(reason="no_grounded_claims", passages=retrieved)

The validation step is the feature. Without it, citations are decoration.

#7. Abstention is a feature, not a failure

Below a retrieval confidence threshold, return "no supported answer found" alongside the closest passages you did find, and let the person judge.

This is a genuinely hard sell to stakeholders, and it is the right call. A confident wrong answer costs more trust than an honest gap — and trust, once spent, takes the whole product down with it. Users forgive a tool that says "I don't know". They abandon one that was confidently wrong about something they could check.

#8. The model will go down

It is the least reliable dependency in the system. Timeouts, one retry, then a circuit breaker after consecutive failures.

When the breaker opens, degrade to retrieval-only: ranked passages, no commentary, and a visible note that analysis is unavailable. The product gets worse and stays useful. The Failure Lab on this site walks through this specific scenario.

// Generation is optional. Retrieval is not.
const passages = await retrieve(query);          // must succeed

const analysis = await breaker
  .run(() => generate(query, passages))
  .catch(() => null);                            // may fail

return { passages, analysis, degraded: analysis === null };

#Where the engineering time actually goes

Ranked by where the hours land, in my experience building these:

1. 01Extraction and chunking — the largest quality lever, and the least discussed.
2. 02Evaluation — a fixed question set with known-good passages, run on every change. Without it you are tuning by vibes.
3. 03Failure handling — timeouts, breakers, degradation, abstention.
4. 04Retrieval tuning — hybrid weighting, k, rerank window.
5. 05Prompting — real, but far smaller than its share of the discourse.

The model is a component you configure. Everything around it is the system you build.

#Try the mechanics

The AI Lab runs a genuine retrieval implementation in your browser — TF-IDF vectors and cosine similarity over a small corpus, with no network call. It is not a neural embedding model, but the mechanics on display are the real ones: a document is a vector, a query is a vector, and relevance is the angle between them.