Leia este artigo em português

The answer was wrong and the team did what every team does: swapped the model for a bigger one. No change. Rewrote the system prompt three times, added “answer only from the provided context,” dropped the temperature to zero. Still wrong. When someone finally printed what had actually been sent to the model, the discussion ended in thirty seconds: the passage holding the answer was not there. The model had been right all along — it faithfully summarized three paragraphs that did not answer the question. Nobody had a generation problem. They had a retrieval problem, and they had spent two weeks tuning the wrong stage.

Three budgets Retrieval is not a single box that works better or worse. It is a funnel with three stages, and each one fails for a different reason and takes a different fix. The first is recall: the right passage has to enter the candidate set. The second is ranking precision: once inside, it has to climb to the top. The third is passage context: at the top or not, it has to answer on its own, away from the document it came from. Almost every “let us improve the RAG” effort I meet in the field is attacking the second stage while the problem lives in the first. Which brings us to the line in the title, and it is not my opinion — it is documented mechanics: a reranker cannot fix what search never retrieved.

What is advanced RAG?

Introductory RAG is a straight line: split the document, generate embeddings, search by similarity, drop the results into the prompt. I have written about that path before, both in the RAG architecture with Azure AI Search and Azure OpenAI and in the PDF-to-index assistant that does not hallucinate. It works in a demo because the corpus is small and the questions are kind.

Advanced RAG is what happens when the corpus grows and the questions stop being kind. It stops being “search by similarity” and becomes a pipeline with named stages: document parsing, chunking, metadata filtering, first-stage retrieval (L1), rank fusion, second-stage reranking (L2), and context assembly. Each stage has its own metric and its own ceiling. And here is the part that changes everything: the stages are sequential and never talk backwards. What L1 did not retrieve, L2 never sees.

This article is the direct sequel to The vector index does not inherit permissions, which dealt with the permission model of the two data access routes. Here the subject is the other axis: retrieval quality. A perfectly governed index that returns the wrong paragraph is still a problem — just a different kind.

The problem it solves

The problem is diagnosis. Without separating the three stages, every RAG failure looks the same — “the agent answered wrong” — and the team treats all of them with the same tool, usually the most visible one.

Three symptoms that look alike and are not:

1. The answer ignores a document you know exists. Recall failure: the passage never entered the candidate set. A reranker will not fix it, a bigger top_k almost never fixes it, and no prompt fixes it.
2. The right document came back in seventh place, and the context window cut at five. Ranking failure. Here the reranker is exactly the right tool.
3. The right passage came back first and the answer is still poor. Passage-context failure: it was cut in the middle of a table, lost its section heading, or says “as described above” about an “above” that stayed in the previous chunk.

The expensive mistake is confusing 1 with 2, and it is structurally induced: the Azure AI Search documentation is explicit in saying that the semantic ranker operates over the top 50 L1 results and does not query the index again. Fifty. If your passage landed in 51st place, no reranking configuration on earth will find it.

How it works — step by step

1. Build the evaluation set before touching any parameter. Databricks is blunt in its retrieval quality page: if you do not have evaluation in place, stop and set it up first, because optimizing without measurement is guesswork. Set the latency target up front too — time to first token in conversational RAG tolerates something under two seconds; a search bar does not.
2. Parse the document before chunking it. A PDF flattened into running text loses tables, headings, and hierarchy. That loss is irreversible: no later stage reconstructs what parsing threw away.
3. Chunk with overlap and respect sentence boundaries. The documented starting point in Azure AI Search is 512 tokens with 25% overlap, which is 128 tokens repeated between neighboring chunks.
4. Write metadata onto every chunk. Document title, section, date, product, business unit. That is what makes filtering possible.
5. Filter before you search. Databricks calls metadata filtering the biggest lever available, able to cut the search space by more than 90%.
6. Use hybrid search by default. Keyword and vector fail in different situations, and fusion covers each one’s blind spot.
7. Turn on second-stage reranking. It is the single largest relevance gain Microsoft documents — and it stays capped at the 50 candidates L1 handed over.
8. Measure retrieval separately from the answer. They are two different scores. Blending them is what produces “we think it improved.”
9. Only then touch the embedding model. Databricks itself puts embedding fine-tuning as the last resort in its ordered list.

Recall: the passage has to enter the candidate set

Everything that decides recall happens before the query exists. It is indexing work, not search work.

Chunking is where recall leaks most quietly. Microsoft’s public numbers, measured as recall@50 with pure vector search, show the effect of size and boundary:

Chunk configuration	Recall@50
512 tokens	42.4
1,024 tokens	37.5
4,096 tokens	36.4
8,191 tokens	34.9
512 tokens, cut at token boundary	40.9
512 tokens, preserving sentences	42.4
512 tokens, sentences + 10% overlap	43.1
512 tokens, sentences + 25% overlap	43.9

Two readings: bigger chunks hurt recall, and overlap is nearly free. The sharpest contrast in the same study is between indexing a whole document as a single vector and chunking it — when the answer sits deep in the document, recall goes from 28.7 to 51.4.

And here is the configuration trap I run into most: the recommendation is 2,000 characters per page, but the Azure AI Search text split skill defaults to 5,000 characters, two and a half times the recommendation. Anyone who does not set it explicitly starts out of range.

{
  "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
  "textSplitMode": "pages",
  "maximumPageLength": 2000,
  "pageOverlapLength": 500
}

If the document has structure — and contracts, standards, and manuals do — it pays to swap the generic splitter for layout parsing, which returns markdown with the heading hierarchy intact and lets you cut by heading depth instead of by character count.

Why vector alone breaks on literal queries

Vector search is good at paraphrase and terrible at literal. When the user types a product code, a contract number, or an error message, semantic similarity has nothing to work with — and Microsoft’s per-query-type numbers are close to embarrassing: on keyword queries, keyword search scores 79.2 NDCG@3 against 11.7 for vector. On exact snippet search, 51.1 against 41.5.

Hybrid solves this by running both and fusing the lists with RRF (Reciprocal Rank Fusion), constant k = 60 — the same constant, by useful coincidence, in Azure AI Search and Databricks. The operational detail almost nobody reads: to appear in the fused result, a document has to rank in the top 50 of at least one subquery. Fusion does not invent candidates; it reconciles two rankings.

It is worth knowing the scales, because they are not comparable to each other: BM25 returns an unbounded @search.score, vector search returns @search.score between 0.333 and 1.00, and RRF returns a value bounded by the number of subqueries.

Precision: the reranker only sees the top fifty

Second-stage reranking is a cross-encoder: instead of comparing two precomputed vectors, it reads the query and the passage together and scores relevance. It is expensive per item, which is why it runs only at the end of the funnel.

It is also, by a wide margin, the largest documented gain. Microsoft’s November 2024 report, with the current semantic ranking model, measures average NDCG across BEIR, MIRACL, and internal sets:

Configuration (embedding text-embedding-ada-002)	Average NDCG	Delta over hybrid
Text only	42.6	—
Vector only	45.1	—
Hybrid	48.4	—
Hybrid + previous semantic ranker	63.1	+14.7
Hybrid + current semantic ranker	71.1	+22.7
Hybrid + query rewriting + current ranker	71.7	+23.3

Three conclusions I would draw from that table before any others:

• Reranking is worth more than swapping the embedding. With text-embedding-3-large at 3,072 dimensions, the final result is 72.8 against 71.7 for ada-002: 1.1 points. If you are debating an embedding migration before turning the reranker on, the order is wrong.
• Storage cost is negotiable. The same table shows text-embedding-3-large truncated to 256 dimensions with binary quantization reaching 71.7 — effectively tied with the full configuration, at an index reduction the documentation puts at up to 28 times.
• Query rewriting adds little on top of hybrid with a reranker (somewhere between half a point and one point). It shines in a different scenario, which I cover below.

Two semantic ranker mechanics that produce silent failure. First: the per-document summary budget is 2,048 tokens, distributed by field order in the semantic configuration — title takes up to 128 tokens, keywords up to 128, and the content field inherits the remainder. Anything past the budget is dropped without warning. If your content field is listed after two long fields, you are reranking a truncated summary.

Second: @search.rerankerScore runs from 0.00 to 4.00 with documented meaning per band, but the documentation itself warns that the distribution shifts with model and infrastructure updates. Do not build business rules on fine-grained thresholds.

Context: the passage has to answer on its own

This is the forgotten budget. A retrieved passage is read outside its document, without the headings that preceded it and without the paragraph that defined the subject. “The term is 30 days” is useless when it was the previous chunk that said which term.

The best-measured public technique for this is Anthropic’s contextual retrieval: before indexing, prepend each chunk with a short summary — 50 to 100 tokens — placing it inside the document, and index that enriched text in both the vector and the BM25 side. Results, as retrieval failure rate (1 − recall@20):

Technique	Failure rate	Reduction
Baseline (embeddings)	5.7%	—
+ contextual embeddings	3.7%	−35%
+ contextual embeddings + contextual BM25	2.9%	−49%
+ reranking	1.9%	−67%

Note the order: the reranker comes last and on top of an already repaired base. Applied alone, it would have far less to work with.

Being honest about the state of the platforms: neither Azure AI Search nor Databricks offers contextual retrieval as a managed feature. The sanctioned path on Azure is the generative prompt skill, which calls chat completions during indexing and became generally available in the 2026-04-01 API — with a hard 30-second per-request limit that is not configurable. On Databricks, the guidance is to enrich the chunk with semantic metadata, inline in the text or in columns passed to the reranker.

When query rewriting and agentic retrieval pay off

Query rewriting generates variations of the user’s question and searches all of them. The gain over hybrid with a reranker is small, but it matters when L1 is text-only, when the query arrives misspelled, when recall is poor, or when the question mixes text and numbers. It is still in preview, depends on the semantic ranker, and is not available in every region — check yours before designing on top of it.

Agentic retrieval is the next step: instead of one query, a planner decomposes the question into several and reconciles the answers. What changed here deserves attention from anyone who wrote code earlier: knowledge agents were renamed to knowledge bases and became generally available in the 2026-04-01 API, but query planning and answer synthesis remain in preview. And there is a breaking billing change: semantic ranking and agentic retrieval are now billed separately, which forces you to declare knowledgeRetrieval=standard explicitly before upgrading to the new API.

Measure retrieval separately from the answer

Here is the habit that separates teams that improve from teams that think they improved. Groundedness measures the answer. NDCG measures retrieval. An answer perfectly grounded in three irrelevant passages scores well on the first and should not score well on the second.

In Microsoft Foundry, the evaluator that matters for this article is Document Retrieval, which is generally available, does not use an LLM as judge, and requires ground truth — the list of documents that should have been retrieved, with a relevance label. It returns NDCG, XDCG, max relevance, fidelity, and the holes indicator, which counts how many retrieved documents carry no label in your set — the metric that exposes an evaluation set that has aged.

from azure.ai.evaluation import DocumentRetrievalEvaluator

evaluator = DocumentRetrievalEvaluator(
    ground_truth_label_min=0,
    ground_truth_label_max=4,
)
result = evaluator(
    retrieval_ground_truth=[{"document_id": "doc-12", "query_relevance_label": 4}],
    retrieved_documents=[{"document_id": "doc-12", "relevance_score": 0.98}],
)

The documentation describes the working method itself: generate results while varying the search algorithm, top_k, and chunk size, then use the evaluator to find which combination yields the best retrieval. That is a parameter sweep, not intuition.

On the Databricks side, agent evaluation was folded into MLflow 3, and the judges that matter here are RetrievalRelevance, RetrievalGroundedness, and RetrievalSufficiency — the last one answering exactly the funnel’s question: is the retrieved context sufficient to support the expected facts?

One honest caveat: neither platform publishes how many test queries are enough. Set your own floor and compare relative improvement between runs, not absolute values against a public benchmark.

What changes on Databricks

Same anatomy, with its own names and defaults. Databricks Vector Search is now Databricks AI Search, and there is a detail that bites anyone assuming Azure behavior: the index uses HNSW with L2 distance, so if you want cosine similarity you have to normalize the embeddings yourself.

The reranker is built in and runs as a second stage over the hybrid result:

from databricks.ai_search import AISearchClient, DatabricksReranker

client = AISearchClient()
results = client.get_index("catalog.schema.index").similarity_search(
    query_text="termination notice period in a supply contract",
    columns=["id", "text", "section"],
    reranker=DatabricksReranker(columns_to_rerank=["text", "section"]),
    num_results=10,
)

Three documented limits that change the design: column order matters and only the first 2,000 characters across them are considered; num_results is the final count and does not control how many candidates get reranked; and default throughput sits around 5 queries per second, taking roughly a second to rerank 50 results — which rules out an interactive search bar. The advertised quality gain sits between 10% and 15%, depending on which documentation page you read.

Production best practices

• Diagnose before optimizing. Replay the question that failed, print the L1 candidates, and check whether the right passage is among them. That two-minute check decides whether the problem is recall or ranking.
• Hybrid by default, pure vector as the exception. It is one configuration line and it covers the entire literal-query class.
• Metadata filtering before any fine-tuning. It is the cheapest lever and the highest impact one.
• Do not enable query rewriting by reflex. Check regional availability and measure — over hybrid with a reranker the gain is marginal.
• Treat the reranker summary budget as real configuration. Field order decides what gets truncated in silence.
• Prefix context onto the chunk when documents are long and hierarchical. Putting the section title in the chunk solves much of the problem with no model involved.
• Quantize when the index grows, and rescore. Default oversampling is 4, and rescoring requires HNSW — exhaustive search cannot rescore.
• Pin the API version before migrating. The billing split between semantic ranking and agentic retrieval is not backward compatible.
• Keep a copy of any number you cite. The reference tables in this article live in blog posts, not in the documentation — the documentation removed the values.

What has to be in place

Foundation — the corpus is parseable and there is a ruler. Documents go through parsing that preserves tables and hierarchy, every chunk carries metadata, and there is a query set with labeled ground truth. Without that, every later change is faith.
It is in place when you can tell, for a question that failed, whether the right passage entered the candidate set or not.

Production with context — the funnel is complete and measured. Hybrid search with metadata filtering at L1, reranking at L2, chunks that stand on their own outside the document, and two separate scores on the dashboard: one for retrieval, one for the answer.
It is in place when a parameter change moves one of the two scores and you know which one.

Scale and efficiency — cost enters the equation without collapsing relevance. Quantization with oversampling and rescoring, a latency budget declared per use case, and a parameter sweep that runs on its own instead of depending on someone with spare time.
It is in place when shrinking the index stopped being a risky decision and became a number in the comparison table.

The order is causal, not chronological: each stage exists because the previous one made the next one measurable.

Frequently asked questions (FAQ)

Does swapping the embedding model fix poor retrieval?

Rarely, and it is expensive to find out. The benchmark shows a little over one NDCG point between ada-002 and text-embedding-3-large when reranking is on. The cheap diagnostic Databricks suggests is comparing two different models: if the results are comparable, the embedding is not your problem.

Does raising top_k not fix a recall failure?

It helps at the margin and charges for it. More candidates means more latency and more reranking cost, and it does not change the fact that the passage may not be in the index in retrievable form. If the passage was badly chunked, it shows up in no top_k at all.

Do I need a reranker if I already use hybrid search?

Yes, and it is the highest-return change on the list. In the current numbers, hybrid goes from 48.4 to 71.1 average NDCG with the semantic ranker enabled.

What chunk size should I use?

Start at 512 tokens with 25% overlap, respecting sentence boundaries. It is worth knowing there is tension between sources: Microsoft’s most recent benchmark was measured with 1,024-token chunks and no overlap, justified by what customers commonly use.

How do I know whether the problem is recall or ranking?

Compare the position of the correct document. If it appears nowhere in L1, it is recall. If it appears below the context cutoff, it is ranking. That is the same distinction separating holes from NDCG in the Foundry evaluator.

Does agentic retrieval replace the rest of the funnel?

No. It decomposes the question and coordinates several searches, but each of those searches is still subject to the same three budgets. Query planning and answer synthesis, on top of that, are still in preview.

Can I compare my NDCG with the published benchmark?

Not directly. The public numbers come from specific datasets, with a specific embedding model and a specific chunk size. Use them to order decisions and track the relative movement of your own measurements.

Conclusion

RAG stopped being a technique and became a pipeline, and pipelines are debugged stage by stage. What I would carry from this into the next architecture review is the sequence: recall is bought at indexing time, precision is bought in the second stage, and context is bought inside the chunk — and none of those currencies converts into the others. In the previous article, about data agents across Databricks, Fabric, and Foundry, the conclusion was that the index does not inherit the table’s permissions. This is the quality version of the same idea: the second stage does not inherit what the first one discarded.

Official references

• Hybrid search overview in Azure AI Search
• How hybrid search ranking works (RRF)
• Relevance and ranking in Azure AI Search
• Semantic ranking
• Query rewriting
• Chunk documents for vector search
• Text split skill
• Document layout skill
• GenAI prompt skill
• Compress vectors using quantization
• What is new in Azure AI Search
• RAG evaluators in Microsoft Foundry
• Raising the bar for RAG excellence: query rewriting and a new ranking model
• Outperforming vector search with hybrid retrieval and reranking
• Retrieval quality in Databricks AI Search
• Query a Databricks AI Search index
• Contextual retrieval (Anthropic)

👉 If you have RAG in production answering almost right — especially if the team already swapped models looking for a fix, start by printing the first-stage candidates.
Want to talk about retrieval, reranking, and RAG evaluation? Reach out on LinkedIn.