Skip to content
ContentLora

    Tip: press / anywhere to search.

    Explainer

    Retrieval-augmented generation (RAG) explained

    Retrieval-augmented generation (RAG) improves a language model's answers by grounding them in information retrieved after the model was trained.[1] A RAG system searches a knowledge base for passages relevant to the question, adds them to the prompt, and asks the model to answer from that material, which also lets it cite sources.[2][3] It does not guarantee accuracy: answers depend on what the search finds, and grounded text can still be wrong if the source is wrong.[4][5]

    Editor reviewedUpdated Explainer21 claims · 100% primaryArtificial intelligence, Frontier AI

    This explainer covers how RAG systems work in general. It does not compare products. For why language models make things up in the first place, see AI hallucinations.

    Why models need retrieval

    A language model’s knowledge is fixed when training ends. Google Cloud says models are limited to their pre-training data, and that RAG addresses this by supplying up-to-date information.[6] The 2020 paper that named the technique noted that providing sources for a model’s answers and updating what it knows were open problems for models that store knowledge only in their parameters.[7]

    Google’s glossary defines RAG as improving a model’s output by grounding it with knowledge retrieved after the model was trained.[1] Among the reasons it lists are giving the model access to knowledge it was not trained on and letting it cite sources.[3] A 2023 survey describes RAG as a response to hallucination, outdated knowledge and untraceable reasoning in language models.[8]

    How a RAG system works

    The basic loop has three steps. In Google’s example, the system searches for data relevant to the user’s question, appends that data to the question, and instructs the model to answer based on it.[2]

    Before any question arrives, the documents are prepared. Anthropic describes splitting a knowledge base into chunks of usually no more than a few hundred tokens, converting each chunk into an embedding, and storing the embeddings in a vector database that can be searched by meaning.[9] When a user asks something, the database returns the chunks most similar to the query.[9]

    The original 2020 RAG models combined a language model with a searchable index of Wikipedia, and their authors found the combination produced more specific, diverse and factual language than a model relying on its parameters alone.[10] A separate 2020 paper showed that retrieval using learned embeddings alone could beat a strong keyword-based BM25 system by 9 to 19 percentage points in top-20 accuracy.[11]

    Better retrieval

    Retrieval quality matters. Google Cloud calls the retrieval step critically important and says it needs good semantic search over a curated knowledge base.[4]

    Embedding search alone can miss exact terms, such as an error code, that keyword matching like BM25 finds, so many systems combine the two in hybrid search and then use a re-ranker to put the most relevant results first.[12] Anthropic says reranking helps ensure only the most relevant chunks reach the model, but adds some latency as an extra step.[13]

    Chunking creates its own problem. Anthropic says individual chunks can lack enough context, which often makes the system fail to find the right passage.[14] In Anthropic’s own 2024 tests, adding a short explanatory note to each chunk before indexing cut the top-20 retrieval failure rate by 35 percent, by 49 percent when combined with keyword search, and by 67 percent with reranking added.[15] The 2023 survey groups such refinements into stages it calls Naive, Advanced and Modular RAG.[8]

    RAG and long context windows

    As context windows have grown to 1 million tokens or more, some material can simply go into the prompt instead; see tokens and context windows.[16] Anthropic says a knowledge base smaller than about 200,000 tokens, roughly 500 pages, can just be included in the prompt with no need for RAG.[17] Google Cloud says RAG remains useful when the material would not fit in a long context window, or to scale up performance, because it cuts the number of tokens sent and so saves time and cost.[18]

    Long prompts have limits of their own. A 2023 study found that models often used information in the middle of a long input less well than information at the beginning or end.[19]

    Limits

    RAG narrows what a model relies on but does not make it infallible. Google’s glossary notes that a grounded model is not always a factual model, because the source material could itself contain mistakes.[5] If retrieval returns the wrong passages, the model is answering from the wrong material.[4] RAG is also different from fine-tuning, which trains the model further on task-specific examples rather than handing it documents at question time.[20] Anthropic describes the basic building block of AI agents as a language model augmented with retrieval, tools and memory.[21]

    Developing in AI & Computing

    Go deeper

    Questions readers ask

    What is retrieval-augmented generation?

    A technique for improving a large language model's output by grounding it in knowledge retrieved after the model was trained.[1]

    Who invented RAG?

    The term comes from a 2020 paper that combined a language model with a searchable index of Wikipedia. Its authors found the RAG models produced more specific, diverse and factual language than a model relying only on its parameters.[10]

    How does a RAG system find the right documents?

    Typically it splits documents into chunks of a few hundred tokens, converts each into an embedding stored in a vector database, and at question time retrieves the chunks most similar to the query. Many systems add keyword search such as BM25 and a re-ranker.[9][12]

    Does RAG stop AI hallucinations?

    It helps but does not guarantee accuracy. The retrieval step has to find relevant information, and Google notes that a grounded model is not always factual because its source material can contain mistakes.[4][5]

    Is RAG still needed with million-token context windows?

    Sometimes. Anthropic says a knowledge base under about 200,000 tokens can simply go in the prompt, while Google Cloud says RAG reduces the number of tokens when the information would not fit in a long context window or to scale up performance.[17][18]

    What is the difference between RAG and fine-tuning?

    RAG changes what information the model sees at the moment it answers, without retraining it. Fine-tuning trains the model further on task-specific examples, changing its parameters.[1][20]

    Sources

    · 21 claims from 5 publishers

    Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.

    1. [1]

      Google's glossary defines retrieval-augmented generation (RAG) as improving a large language model's output by grounding it with knowledge retrieved after the model was trained. confirmedas of 2026-11-14

      • Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
    2. [2]

      In Google's example, a RAG system searches for data relevant to the user's query, appends it to the query, and then asks the model to answer based on the appended data. confirmedas of 2026-11-14

      • Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
      • Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
    3. [3]

      Google lists common reasons to use RAG, including giving the model knowledge it was not trained on and letting it cite sources. confirmedas of 2026-11-14

      • Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
      • Machine Learning Glossary · Google for Developers · Primary source · Glossary, retrieval-augmented generation (retrieved 2026-11-07)
    4. [4]

      Google Cloud calls the retrieval step critically important and says it needs good semantic search over a curated knowledge base so the retrieved information is relevant. confirmedas of 2026-11-14

    5. [5]

      Google's glossary notes that a model grounded on specific source material is not always factual, because the source itself could contain mistakes. confirmedas of 2026-11-07

    6. [6]

      Google Cloud says language models are limited to their pre-training data and that RAG addresses this by supplying up-to-date information. confirmedas of 2026-11-14

    7. [7]

      The 2020 paper that named RAG said providing provenance for a model's decisions and updating its world knowledge were open problems for models that store knowledge only in their parameters. confirmedas of 2020-05-22

    8. [8]

      A 2023 survey describes RAG as a response to hallucination, outdated knowledge and untraceable reasoning in language models, and groups its development into Naive, Advanced and Modular RAG. confirmedas of 2023-12-18

    9. [9]

      Anthropic describes a typical RAG pipeline as splitting a knowledge base into chunks of usually no more than a few hundred tokens, converting them into embeddings stored in a vector database, and at query time finding the chunks most similar to the query. confirmedas of 2024-09-19

    10. [10]

      Retrieval-augmented generation (RAG), introduced in a 2020 paper, combines a language model with a searchable index of documents such as Wikipedia; its authors found RAG models generated more specific, diverse and factual language than a model relying only on its parameters. confirmedas of 2020-05-22

    11. [11]

      A 2020 paper showed that passage retrieval for question answering could use learned dense embeddings alone, beating a strong BM25 keyword system by 9 to 19 percentage points in top-20 retrieval accuracy. confirmedas of 2020-04-10

    12. [12]

      Embedding search can miss exact terms such as an error code that keyword matching like BM25 finds, so many RAG systems combine the two, called hybrid search, and add a re-ranker to put the most relevant results first. confirmedas of 2026-11-14

    13. [13]

      Anthropic says reranking filters results so that only the most relevant chunks reach the model, but adds some latency because it is an extra step at runtime. confirmedas of 2024-09-19

    14. [14]

      Anthropic says splitting documents into small chunks can cause problems when individual chunks lack enough context, which often makes a RAG system fail to retrieve the relevant information. confirmedas of 2024-09-19

    15. [15]

      In Anthropic's own 2024 tests, adding explanatory context to each chunk cut the top-20 retrieval failure rate by 35 percent with embeddings, 49 percent when combined with contextual BM25, and 67 percent with reranking added. confirmedas of 2024-09-19

    16. [16]

      As of November 2026, Anthropic lists a 1-million-token context window for most of its current Claude models, and Google says many Gemini models have context windows of 1 million or more tokens. confirmedas of 2026-11-14

    17. [17]

      Anthropic says a knowledge base smaller than 200,000 tokens, about 500 pages, can simply be included in the prompt with no need for RAG. confirmedas of 2024-09-19

    18. [18]

      Google Cloud says that when more information is needed than fits in a long context window, or to scale up performance, a RAG approach reduces the number of tokens, saving time and cost. confirmedas of 2026-11-14

    19. [19]

      A 2023 study found that language models often performed best when relevant information was at the beginning or end of a long input and significantly worse when it was in the middle, even for models built for long contexts. confirmedas of 2023-07-06

    20. [20]

      Google says fine-tuning trains on examples specific to the task and can sometimes use just a few hundred or a few thousand training examples. confirmedas of 2026-11-14

    21. [21]

      Anthropic describes the basic building block of agentic systems as a language model augmented with retrieval, tools and memory. confirmedas of 2024-12-19

    Revision history (1)
    1. Page created.

    Created Nov 14, 2026. Last reviewed by an editor on Nov 14, 2026. Next scheduled review: May 14, 2027.

    Cite this page

    "Retrieval-augmented generation (RAG) explained." ContentLora, updated Nov 14, 2026. https://contentlora.com/explain/retrieval-augmented-generation-explained

    Spotted an error? Suggest a correction or emailcorrections@contentlora.com.