Explainer
Retrieval-augmented generation (RAG) explained
Retrieval-augmented generation (RAG) improves a language model's answers by grounding them in information retrieved after the model was trained.[1] A RAG system searches a knowledge base for passages relevant to the question, adds them to the prompt, and asks the model to answer from that material, which also lets it cite sources.[2][3] It does not guarantee accuracy: answers depend on what the search finds, and grounded text can still be wrong if the source is wrong.[4][5]
This explainer covers how RAG systems work in general. It does not compare products. For why language models make things up in the first place, see AI hallucinations.
Why models need retrieval
A language model’s knowledge is fixed when training ends. Google Cloud says models are limited to their pre-training data, and that RAG addresses this by supplying up-to-date information.[6] The 2020 paper that named the technique noted that providing sources for a model’s answers and updating what it knows were open problems for models that store knowledge only in their parameters.[7]
Google’s glossary defines RAG as improving a model’s output by grounding it with knowledge retrieved after the model was trained.[1] Among the reasons it lists are giving the model access to knowledge it was not trained on and letting it cite sources.[3] A 2023 survey describes RAG as a response to hallucination, outdated knowledge and untraceable reasoning in language models.[8]
How a RAG system works
The basic loop has three steps. In Google’s example, the system searches for data relevant to the user’s question, appends that data to the question, and instructs the model to answer based on it.[2]
Before any question arrives, the documents are prepared. Anthropic describes splitting a knowledge base into chunks of usually no more than a few hundred tokens, converting each chunk into an embedding, and storing the embeddings in a vector database that can be searched by meaning.[9] When a user asks something, the database returns the chunks most similar to the query.[9]
The original 2020 RAG models combined a language model with a searchable index of Wikipedia, and their authors found the combination produced more specific, diverse and factual language than a model relying on its parameters alone.[10] A separate 2020 paper showed that retrieval using learned embeddings alone could beat a strong keyword-based BM25 system by 9 to 19 percentage points in top-20 accuracy.[11]
Better retrieval
Retrieval quality matters. Google Cloud calls the retrieval step critically important and says it needs good semantic search over a curated knowledge base.[4]
Embedding search alone can miss exact terms, such as an error code, that keyword matching like BM25 finds, so many systems combine the two in hybrid search and then use a re-ranker to put the most relevant results first.[12] Anthropic says reranking helps ensure only the most relevant chunks reach the model, but adds some latency as an extra step.[13]
Chunking creates its own problem. Anthropic says individual chunks can lack enough context, which often makes the system fail to find the right passage.[14] In Anthropic’s own 2024 tests, adding a short explanatory note to each chunk before indexing cut the top-20 retrieval failure rate by 35 percent, by 49 percent when combined with keyword search, and by 67 percent with reranking added.[15] The 2023 survey groups such refinements into stages it calls Naive, Advanced and Modular RAG.[8]
RAG and long context windows
As context windows have grown to 1 million tokens or more, some material can simply go into the prompt instead; see tokens and context windows.[16] Anthropic says a knowledge base smaller than about 200,000 tokens, roughly 500 pages, can just be included in the prompt with no need for RAG.[17] Google Cloud says RAG remains useful when the material would not fit in a long context window, or to scale up performance, because it cuts the number of tokens sent and so saves time and cost.[18]
Long prompts have limits of their own. A 2023 study found that models often used information in the middle of a long input less well than information at the beginning or end.[19]
Limits
RAG narrows what a model relies on but does not make it infallible. Google’s glossary notes that a grounded model is not always a factual model, because the source material could itself contain mistakes.[5] If retrieval returns the wrong passages, the model is answering from the wrong material.[4] RAG is also different from fine-tuning, which trains the model further on task-specific examples rather than handing it documents at question time.[20] Anthropic describes the basic building block of AI agents as a language model augmented with retrieval, tools and memory.[21]
Read next
Artificial intelligence: a crash course
A sourced crash course on AI in 2026: how models work, open vs closed weights, the frontier labs, jobs, copyright, state laws and safety.
Questions readers ask
What is retrieval-augmented generation?
A technique for improving a large language model's output by grounding it in knowledge retrieved after the model was trained.[1]
Who invented RAG?
The term comes from a 2020 paper that combined a language model with a searchable index of Wikipedia. Its authors found the RAG models produced more specific, diverse and factual language than a model relying only on its parameters.[10]
How does a RAG system find the right documents?
Typically it splits documents into chunks of a few hundred tokens, converts each into an embedding stored in a vector database, and at question time retrieves the chunks most similar to the query. Many systems add keyword search such as BM25 and a re-ranker.[9][12]
Does RAG stop AI hallucinations?
It helps but does not guarantee accuracy. The retrieval step has to find relevant information, and Google notes that a grounded model is not always factual because its source material can contain mistakes.[4][5]
Is RAG still needed with million-token context windows?
Sometimes. Anthropic says a knowledge base under about 200,000 tokens can simply go in the prompt, while Google Cloud says RAG reduces the number of tokens when the information would not fit in a long context window or to scale up performance.[17][18]
Sources
· 21 claims from 5 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
Google's glossary defines retrieval-augmented generation (RAG) as improving a large language model's output by grounding it with knowledge retrieved after the model was trained. confirmedas of 2026-11-14
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
- [2]
In Google's example, a RAG system searches for data relevant to the user's query, appends it to the query, and then asks the model to answer based on the appended data. confirmedas of 2026-11-14
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
- [3]
Google lists common reasons to use RAG, including giving the model knowledge it was not trained on and letting it cite sources. confirmedas of 2026-11-14
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
- [4]
Google Cloud calls the retrieval step critically important and says it needs good semantic search over a curated knowledge base so the retrieved information is relevant. confirmedas of 2026-11-14
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Relevance, accuracy, and quality (retrieved 2026-11-14)
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Relevance, accuracy, and quality (retrieved 2026-11-14)
- [5]
Google's glossary notes that a model grounded on specific source material is not always factual, because the source itself could contain mistakes. confirmedas of 2026-11-07
- Machine Learning Glossary · Google for Developers · Primary source · Glossary, groundedness (retrieved 2026-11-07)
- [6]
Google Cloud says language models are limited to their pre-training data and that RAG addresses this by supplying up-to-date information. confirmedas of 2026-11-14
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Why use RAG? (retrieved 2026-11-14)
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Why use RAG? (retrieved 2026-11-14)
- [7]
The 2020 paper that named RAG said providing provenance for a model's decisions and updating its world knowledge were open problems for models that store knowledge only in their parameters. confirmedas of 2020-05-22
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv · Primary source · 2020-05-22 · Abstract (retrieved 2026-11-07)
- [8]
A 2023 survey describes RAG as a response to hallucination, outdated knowledge and untraceable reasoning in language models, and groups its development into Naive, Advanced and Modular RAG. confirmedas of 2023-12-18
- Retrieval-Augmented Generation for Large Language Models: A Survey · arXiv · Primary source · 2023-12-18 · Abstract (retrieved 2026-11-14)
- Retrieval-Augmented Generation for Large Language Models: A Survey · arXiv · Primary source · 2023-12-18 · Abstract (retrieved 2026-11-14)
- [9]
Anthropic describes a typical RAG pipeline as splitting a knowledge base into chunks of usually no more than a few hundred tokens, converting them into embeddings stored in a vector database, and at query time finding the chunks most similar to the query. confirmedas of 2024-09-19
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · A primer on RAG (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · A primer on RAG (retrieved 2026-11-14)
- [10]
Retrieval-augmented generation (RAG), introduced in a 2020 paper, combines a language model with a searchable index of documents such as Wikipedia; its authors found RAG models generated more specific, diverse and factual language than a model relying only on its parameters. confirmedas of 2020-05-22
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv · Primary source · 2020-05-22 · Abstract (retrieved 2026-11-07)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv · Primary source · 2020-05-22 · Abstract (retrieved 2026-11-07)
- [11]
A 2020 paper showed that passage retrieval for question answering could use learned dense embeddings alone, beating a strong BM25 keyword system by 9 to 19 percentage points in top-20 retrieval accuracy. confirmedas of 2020-04-10
- Dense Passage Retrieval for Open-Domain Question Answering · arXiv · Primary source · 2020-04-10 · Abstract (retrieved 2026-11-14)
- Dense Passage Retrieval for Open-Domain Question Answering · arXiv · Primary source · 2020-04-10 · Abstract (retrieved 2026-11-14)
- [12]
Embedding search can miss exact terms such as an error code that keyword matching like BM25 finds, so many RAG systems combine the two, called hybrid search, and add a re-ranker to put the most relevant results first. confirmedas of 2026-11-14
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · A primer on RAG (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · A primer on RAG (retrieved 2026-11-14)
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Search with vector databases and relevancy re-rankers (retrieved 2026-11-14)
- [13]
Anthropic says reranking filters results so that only the most relevant chunks reach the model, but adds some latency because it is an extra step at runtime. confirmedas of 2024-09-19
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Further boosting performance with Reranking (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Cost and latency considerations (retrieved 2026-11-14)
- [14]
Anthropic says splitting documents into small chunks can cause problems when individual chunks lack enough context, which often makes a RAG system fail to retrieve the relevant information. confirmedas of 2024-09-19
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · The context conundrum in traditional RAG (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Introduction (retrieved 2026-11-14)
- [15]
In Anthropic's own 2024 tests, adding explanatory context to each chunk cut the top-20 retrieval failure rate by 35 percent with embeddings, 49 percent when combined with contextual BM25, and 67 percent with reranking added. confirmedas of 2024-09-19
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Performance improvements (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Performance improvements (retrieved 2026-11-14)
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · Further boosting performance with Reranking (retrieved 2026-11-14)
- [16]
As of November 2026, Anthropic lists a 1-million-token context window for most of its current Claude models, and Google says many Gemini models have context windows of 1 million or more tokens. confirmedas of 2026-11-14
- Context windows (Claude API documentation) · Anthropic · Primary source · Context window sizes by model (retrieved 2026-11-14)
- Long context (Gemini API) · Google AI for Developers · Primary source · Long context (retrieved 2026-11-14)
- [17]
Anthropic says a knowledge base smaller than 200,000 tokens, about 500 pages, can simply be included in the prompt with no need for RAG. confirmedas of 2024-09-19
- Introducing Contextual Retrieval · Anthropic · Primary source · 2024-09-19 · A note on simply using a longer prompt (retrieved 2026-11-14)
- [18]
Google Cloud says that when more information is needed than fits in a long context window, or to scale up performance, a RAG approach reduces the number of tokens, saving time and cost. confirmedas of 2026-11-14
- What is Retrieval-Augmented Generation (RAG)? · Google Cloud · Primary source · Grounding with long context windows (retrieved 2026-11-14)
- [19]
A 2023 study found that language models often performed best when relevant information was at the beginning or end of a long input and significantly worse when it was in the middle, even for models built for long contexts. confirmedas of 2023-07-06
- Lost in the Middle: How Language Models Use Long Contexts · arXiv · Primary source · 2023-07-06 · Abstract (retrieved 2026-11-14)
- [20]
Google says fine-tuning trains on examples specific to the task and can sometimes use just a few hundred or a few thousand training examples. confirmedas of 2026-11-14
- LLMs: Fine-tuning, distillation, and prompt engineering (Machine Learning Crash Course) · Google for Developers · Primary source · Fine-tuning (retrieved 2026-11-14)
- [21]
Anthropic describes the basic building block of agentic systems as a language model augmented with retrieval, tools and memory. confirmedas of 2024-12-19
- Building effective agents · Anthropic · Primary source · 2024-12-19 · The augmented LLM section (retrieved 2026-10-10)
Revision history (1)
- Page created.
Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: May 14, 2027.
Cite this page
"Retrieval-augmented generation (RAG) explained." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/retrieval-augmented-generation-explained
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.