What Is RAG (Retrieval-Augmented Generation)? (2026)

A RAG pipeline has two phases. First, ingestion (done ahead of time): you split your source documents into smaller passages (chunking), convert each into an embedding — a numeric vector capturing its meaning — and store those vectors in a database.

Second, retrieval and generation (at query time): you embed the user's question, find the most semantically similar passages via vector search, and insert those passages into the prompt alongside the question. The model then generates an answer grounded in the supplied text, ideally with citations back to the source.

The whole point is that the model answers from evidence you placed in its context window, not from whatever it happened to memorize during training. That makes answers traceable — you can show which passage a claim came from — and keeps them current without retraining.

These solve different problems. RAG injects knowledge — facts, documents, policies — into the prompt at query time. Fine-tuning adjusts the model's weights to change behavior, style, or format. A useful rule of thumb: RAG is for what the model should know; fine-tuning is for how the model should act.

Reach for RAG when your knowledge changes often, is large or private, or must be cited — product docs, support tickets, contracts, an internal wiki. You can update the knowledge base instantly without touching the model. Reach for fine-tuning when you need a consistent output format, tone, or a narrow task behavior that prompting alone cannot reliably enforce. The two are not mutually exclusive — many production systems fine-tune for behavior and use RAG for facts.

We compare the tradeoffs, costs, and failure modes in depth in RAG vs fine-tuning: when each wins.

Use RAG when: Knowledge changes often, is large or private, or must be cited; you need answers grounded in specific documents; you want to update facts without retraining.
Use fine-tuning when: You need consistent format, tone, or a narrow behavior; the task is stable; prompting alone can't reliably enforce the output shape you require.

RAG changes prompt structure. Your prompt now has three parts: instructions, the retrieved context, and the user's question. Keep them clearly separated with delimiters (headings, XML-style tags, or triple backticks) so the model can tell its instructions from the supplied data — this also reduces the risk of injection from untrusted documents.

Tell the model to answer only from the provided context and to say when the answer is not there. A fallback instruction such as 'If the context does not contain the answer, say you do not know' is what stops a grounded system from quietly inventing facts. Asking for citations back to the passages used makes answers verifiable.

Here is a minimal grounded-answer prompt skeleton:

``` You are a support assistant. Answer ONLY using the context below. If the answer is not in the context, say "I don't have that information." Cite the source passage number for each claim. Context: [1] {retrieved_passage_1} [2] {retrieved_passage_2} Question: {user_question} ```

Most RAG problems are retrieval problems, not generation problems. If the right passage never gets fetched, the model cannot answer correctly no matter how good the prompt is — so retrieval quality (chunking strategy, embedding model, and how many passages you pull) is where most tuning effort goes.

Other frequent issues: chunks too large to be precise or too small to carry meaning; too much retrieved text crowding out the question or burying the relevant passage in a long context; and trusting retrieved content blindly, which opens the door to prompt injection if your sources are untrusted. Grounding instructions, citations, and tight retrieval are the standard defenses.