From PDF to Searchable Index: An Assistant That Does Not Hallucinate

From PDF to Searchable Index: An Assistant That Does Not Hallucinate

One of the biggest fears when adopting generative AI is hallucination: the model gives an answer that sounds convincing but is wrong. The good news is that you can drastically reduce this by grounding the model in your documents. I will show the path from PDF to searchable index — the foundation of a reliable assistant.

Why does the model hallucinate?

On its own, the model answers from what it “memorized” during training — which may be outdated or may not cover your business. Without a source to rely on, it “fills in the gaps” and invents. The solution is to give it the right answer before it responds.

The strategy: RAG + grounding

RAG retrieves relevant passages from your documents and injects them into the prompt. Grounding is the practice of anchoring the answer in real sources and measuring how much it actually relies on them (groundedness). Together, they turn “guesswork” into an “answer with a source”.

Pipeline from PDF to index: Document Intelligence performs OCR, the text becomes chunks and embeddings, goes to Azure AI Search, and the grounded LLM answers by citing the source.
From PDF to cited answer: extract well, index by meaning, and ground the answer in the sources.

Step by step: from PDF to index

  1. Extract the text: use Document Intelligence (OCR + layout) to read PDFs, including tables and scans.
  2. Break into chunks: split the text into pieces of ~300–500 tokens, preserving the context of each section.
  3. Generate embeddings: convert each chunk into a vector with an embeddings model.
  4. Index: store the vectors in Azure AI Search (vector index). This is what enables search by meaning.
  5. Retrieve + answer: for each question, retrieve the closest chunks and ask the model to answer only based on them.
  6. Cite the source: show which document and page the answer came from.

How to truly avoid hallucination

  • Firm instruction: “answer only based on the provided passages; if there is no basis, say you do not know”.
  • Measure groundedness: use evaluators to score how much the answer relies on the sources.
  • Show citations: transparency discourages invented answers and builds trust.
  • Continuous evaluation: monitor in production, not only during testing.

The model that answers makes a difference in the hallucination rate: replace the older GPT-4/GPT-4o with a more recent one from Azure AI Foundry — GPT-4.1, GPT-5, or o4-mini — which follow the instruction to “answer only based on the passages” better and cite sources more consistently.

Frequently Asked Questions (FAQ)

What is the best chunk size?

There is no magic number. Start with 300–500 tokens while preserving the context of each section, and adjust by measuring answer quality.

Can I use RAG with scanned documents?

Yes. Document Intelligence performs OCR + layout, extracting text even from scanned PDFs and tables before indexing.

How do I know whether the assistant is reliable?

Measure groundedness (how much the answer relies on sources) with evaluators, show citations, and perform continuous evaluation in production.

Conclusion

An “assistant that does not hallucinate” is not magic — it is engineering: extract well, index by meaning, retrieve the right context, and ground the answer. With RAG and grounding in Azure, you deliver AI the company can trust.

👉 Want to learn more about this and other topics? Visit my channel.

Leave a Reply