What is RAG?
Retrieval-Augmented Generation (RAG) is an architecture for AI systems that combines a language model’s ability to generate natural language with a retrieval mechanism that fetches relevant documents at query time. Instead of relying solely on the knowledge baked into the model during training — which has a cutoff date and can’t include your proprietary data — a RAG system searches a live document store first, then passes the retrieved content into the model’s context window alongside the user’s question.
The result is a system that can answer questions using current, specific, and private information without needing to retrain the underlying model. Ask it about your Q3 pipeline, your internal HR policies, or the specs of a product your company launched last week, and it draws from your actual documents rather than making something up.
RAG was formally described in a 2020 paper from Meta AI, but the concept is now foundational to virtually every serious enterprise AI deployment. When companies talk about building a “knowledge base” that their AI assistant can reference, they’re describing RAG.
How RAG Works
The architecture has three main components:
- Document store and embeddings: Your source documents — PDFs, internal wikis, CRM notes, product specs — are split into chunks and converted into vector embeddings (numerical representations of semantic meaning). These are stored in a vector database.
- Retrieval: When a user asks a question, that query is also converted into an embedding. The system computes similarity between the query embedding and all document embeddings, then retrieves the most relevant chunks — typically the top 3–10.
- Generation: The retrieved chunks are injected into the prompt alongside the user’s original question. The LLM then generates an answer grounded in those specific documents, rather than from general training data alone.
Good RAG implementations also include citation handling (showing which document each claim came from), re-ranking steps to improve relevance, and metadata filtering to scope retrieval to relevant subsets of the document store.
RAG vs Fine-Tuning
Fine-tuning and RAG are often presented as competing approaches to teaching an AI about your specific domain, but they solve different problems. Fine-tuning adjusts the model’s weights by training on domain-specific examples — it changes how the model thinks, not just what it knows. RAG adjusts what information the model has access to at query time — it doesn’t change the model itself.
When to use which:
- Use RAG when: You need the system to reference current, frequently updated, or proprietary documents. You want to be able to trace answers to sources. You need to update the knowledge base without an ML pipeline.
- Use fine-tuning when: You want the model to adopt a specific tone, format, or reasoning style. You’re training on examples of correct behavior rather than facts. The task requires a different output structure than the base model produces.
- Use both when: You need a model that both reasons in a domain-specific way and can access current documents. Fine-tune for behavior, RAG for knowledge.
For most enterprise applications, RAG is the right starting point. It’s faster to implement, easier to update, and more interpretable than fine-tuning — and for most use cases, grounding the model in good source documents does more for accuracy than behavioral fine-tuning would.
When to Use RAG
RAG is the right architecture when some combination of these is true: your AI application needs to reference proprietary or recent information, you want to reduce hallucinations by anchoring answers to specific sources, you need to be able to audit what the AI said and why, or your knowledge base changes frequently enough that retraining would be impractical.
Common enterprise RAG applications include internal knowledge bases (“ask our policies anything”), customer support augmentation (routing and answering tickets with product documentation), sales enablement (surfacing relevant case studies or product specs during a conversation), and legal and compliance (searching contracts or regulatory documents in natural language).
The main limitation of RAG is that retrieval quality determines answer quality. A poorly chunked document store, weak embeddings, or a mismatch between how users phrase questions and how documents are structured will produce irrelevant retrievals — and the LLM will generate plausible-sounding answers based on bad inputs. RAG is not a shortcut around document quality; it surfaces what’s in your knowledge base, good and bad.