If you work with generative AI, you have already heard of RAG. But what is RAG in practice — and why has it become the standard for making language models answer questions about your data without making things up? In this guide, I explain the concept and show how to build a RAG assistant in Azure AI Foundry.

What is RAG?

RAG (Retrieval-Augmented Generation) is a technique that retrieves relevant excerpts from a knowledge base and injects them into the model’s prompt before it answers. Instead of relying only on what the model “memorized” during training, you ground it in your documents — manuals, policies, contracts, FAQs.

The result: answers that are up to date, specific to your business, and much less prone to hallucination (when the model invents something plausible but false).

How RAG works (in 4 steps)

1. Ingestion: your documents are read (OCR/layout), split into pieces (chunks), and transformed into embeddings — numerical vectors that represent the meaning of the text.
2. Indexing: the embeddings go into a vector index (e.g., Azure AI Search). This is what makes it possible to search by meaning, not by exact word.
3. Retrieval: for each question, the system searches the index for the excerpts most similar to it.
4. Generation: those excerpts are appended to the prompt, and the model answers based on them, citing the source.

Building RAG in Azure AI Foundry — step by step

1. Create a Hub + Project in Azure AI Foundry and choose, from the catalog, a language model (today there are newer and more capable options than GPT-4, such as GPT-4.1, the GPT-5 family, or the o3/o4-mini reasoning models) and an embeddings model (e.g., text-embedding-3-large).
2. Ingest the documents: use Document Intelligence to extract text from PDFs, split it into chunks, and generate embeddings for each one.
3. Index: store the embeddings in Azure AI Search as a vector index.
4. Connect the model to search: configure the “chat with your data” pattern, pointing the model to the index.
5. Grounding: instruct the model to answer only based on the retrieved excerpts and to say “I don’t know” when there is no supporting basis.
6. Evaluate: use Foundry evaluators (groundedness, relevance) before going to production.

2026 update GPT-4o was a milestone, but it has already been surpassed: prefer more recent models from the Foundry catalog — GPT-4.1 and GPT-5 for general quality, or o3/o4-mini when you need step-by-step reasoning and better adherence to the instruction to “answer only based on the sources.” Changing the model is just one parameter; the rest of the RAG architecture stays the same.

Best practices that separate good RAG from bad RAG

• Smart chunking: chunks should be neither too large (diluting the context) nor too small (losing meaning). Start with 300–500 tokens and adjust.
• Cite sources: show which document the answer came from — this builds trust and makes auditing easier.
• Governance (RAI): content filters, continuous evaluation, and Managed Identity — no secrets in code.
• Measure groundedness: this is the metric that tells how much the answer is actually supported by the sources.

Frequently asked questions (FAQ)

Does RAG replace fine-tuning?

Not necessarily. RAG is ideal when knowledge changes frequently or is company-specific; fine-tuning adjusts the model’s behavior/style. Many solutions use both together.

Does RAG completely eliminate hallucination?

It does not eliminate it, but it reduces it significantly. Combined with grounding, source citations, and firm instructions (“answer only based on the excerpts”), the risk drops considerably.

Do I need Azure to do RAG?

No — the pattern is agnostic. But Azure AI Foundry makes it easier because it brings ingestion, vector index, model, and evaluation together in one place.

Conclusion

RAG is currently the most direct path to bring reliable generative AI into the enterprise. With Azure AI Foundry, you have ingestion, vector index, model, and evaluation in one place.

👉 Want to see this in practice? Follow the videos on my channel.