Azure AI Search RAG Tutorial: Build RAG on Azure with Python

On this page
Azure AI Search is Microsoft's managed search service, and on Azure it is the standard building block for the retrieval half of RAG (Retrieval-Augmented Generation). It extracts text from your files, splits it into chunks, embeds them with an Azure OpenAI model, and answers queries with a combination of keyword search, vector search and a reranker. A chat model in Microsoft Foundry then writes the answer from the chunks it gets back.
This tutorial builds that pipeline end to end with the Python SDK
(azure-search-documents 12.0.0) and tests it on a real document: the
144-page EU AI Act. The final setup answered all five test questions
correctly, with article references, in about 1.2 seconds per answer. It
took two fixes to get there, and both are problems you will run into with
your own documents: the Basic tier quietly cut off the last 15 % of the PDF,
and the default chunk size missed a question about application dates. We
tested on 6 November 2026; prices and region availability are from the same
day.
How RAG with Azure AI Search works
RAG has two phases. At indexing time, documents are split into chunks and each chunk is stored with an embedding. At query time, the system finds the chunks that best match the question and passes them to a language model as context. Azure AI Search covers the first phase completely and the search part of the second.
RAG pipeline with Azure AI Search: indexer and skillset fill the index, a hybrid query with semantic ranker feeds gpt-5.4-mini
These are the objects you create in Azure AI Search:
| Object | What it does |
|---|---|
| Data source | Connection to Blob Storage, SharePoint, OneLake, Azure SQL, Cosmos DB and others |
| Skillset | Processing steps during indexing, here the Text Split skill (chunking) and the Azure OpenAI Embedding skill |
| Index | The searchable store: text fields for keyword search, vector fields for similarity search |
| Index projection | Writes each chunk as its own search document instead of one document per file |
| Indexer | Runs the pipeline on demand or on a schedule and only processes new or changed files |
| Vectorizer | Embeds the user's question at query time with the same model, so your code sends plain text |
Microsoft calls the combination of skillset, index projection and vectorizer "integrated vectorization". Your application never computes an embedding itself.
At query time, a hybrid query runs a BM25 keyword search and a vector search (HNSW) in parallel and merges both result lists with Reciprocal Rank Fusion (RRF). The semantic ranker then reranks the top 50 results with a multilingual model derived from Bing. The top five chunks go to the chat model.
Classic RAG, agentic retrieval and Foundry IQ
Microsoft's RAG overview now describes two approaches. "Classic RAG" is the pipeline above: your code sends one hybrid query and calls the model. "Agentic retrieval" lets an LLM plan several subqueries, runs them in parallel and returns merged results. Its resource is called a knowledge base, and Foundry IQ is the name for knowledge bases used by agents in the Microsoft Foundry portal.
The status is mixed. Knowledge bases with minimal, extractive retrieval have
been generally available since REST API 2026-04-01. LLM query planning,
answer synthesis and the reasoning effort levels low and medium are still
in preview (2026-08-01-preview), and the portal shows all agentic retrieval
features in preview mode only. We test both further down.
If you used "Azure OpenAI On Your Data" (the data_sources parameter in the
chat completions API, or "Add your data" in the playground): Microsoft
retired it on 14 October
2026.
Existing search indexes keep working. You now call the search service
yourself, as shown below, or move to a knowledge base.
Choose the tier and region
For data residency in the DACH region, the region matters as much as the tier. The region list (last updated 3 November 2026) marks Sweden Central, West Europe, Germany West Central and North Europe as "high demand", which "prevents the creation of new search services". Existing services there keep running. France Central and Switzerland North accept new services, and all six regions support the semantic ranker and agentic retrieval.
The chat and embedding models can run in a different region than the search
service. We checked model availability with az cognitiveservices model list
on 6 November 2026:
| Region | Search service | gpt-5.4-mini | text-embedding-3-small |
|---|---|---|---|
| France Central | new services possible | DataZoneStandard, GlobalStandard | DataZoneStandard, GlobalStandard |
| Switzerland North | new services possible | DataZoneStandard, GlobalStandard | Standard (regional), GlobalStandard |
| Germany West Central | high demand, no new services | DataZoneStandard, GlobalStandard | DataZoneStandard, GlobalStandard |
| Sweden Central | high demand, no new services | DataZoneStandard, GlobalStandard | DataZoneStandard, GlobalStandard |
We used France Central for everything and deployed both models as
DataZoneStandard. According to the deployment types
page,
the service then processes data only within the EU Data Boundary (which can
include EFTA countries such as Switzerland and Norway), and data at rest
stays in the resource's geography. A GlobalStandard deployment is about
9 % cheaper but "may be processed in any Azure region".
For the tier, Free is fine for a first look, but each subscription gets only one free service, with 50 MB of storage, three indexes and indexer runs capped at three minutes. Basic is the smallest tier for real documents. The import quickstart also requires Basic or higher for managed identities, which we use for every connection.
Set up the Azure resources
You need an Azure subscription, the Azure CLI and Python 3.10 or later. These are the commands we ran; replace the placeholders with your own values:
--auth-options aadOrApiKey enables Microsoft Entra ID (RBAC) next to the
API keys, --identity-type SystemAssigned gives the search service its own
managed identity, and --semantic-search free activates the free plan of the
semantic ranker. The AIServices account is a Microsoft Foundry resource; it
needs a custom subdomain so Azure AI Search can call it. Next, deploy the two
models:
We picked gpt-5.4-mini because Azure already lists gpt-4.1-mini as
"Legacy" (retirement 14 April 2027). The capacity is in thousands of tokens
per minute.
Then assign the roles. The search service's managed identity reads the blobs and calls the embedding model; your own user account manages the index and calls the chat model. No keys end up in code or in the index definition.
Upload the document
Our test document is the EU AI Act, Regulation (EU) 2024/1689 as an English PDF from EUR-Lex (144 pages, 2.6 MB). The EU allows reuse of EUR-Lex documents with attribution. Its long articles, cross-references and annexes make it a good stand-in for contracts, standards and manuals.
The Basic tier only extracts the first 512,000 characters of each file (more on that under pitfalls), so we split the PDF into three parts of 48 pages each before uploading:
Install the Python packages
We used azure-search-documents 12.0.0 (released 1 May 2026, targets REST
API 2026-04-01), azure-identity 1.27.0 and openai 3.46.0. The scripts
read three environment variables:
DefaultAzureCredential signs in with your Azure CLI login. If your account
can see several tenants, run az account set --subscription <id> first: the
CLI credential requests tokens for the tenant of the default subscription,
and a token from the wrong tenant fails with an authentication error.
Step 1: create the index and the indexing pipeline
This script creates the data source, the index, the skillset and the indexer. The indexer starts as soon as it is created; the script waits until the run has finished.
A few details are easy to get wrong:
- The key field
chunk_idneeds thekeywordanalyzer, otherwise index projections cannot write the generated chunk IDs. - The vectorizer in the index and the embedding skill must point to the same model with the same dimensions. Otherwise queries and chunks end up in different vector spaces.
en.microsoftis the language analyzer for keyword search. For German documents usede.microsoft.- No connection string contains a key. The data source uses
ResourceId=...and both the skill and the vectorizer authenticate with the search service's managed identity.
The run on the three parts took 73 seconds and produced 809 chunks. The index uses 18.7 MB, 5.0 MB of which is the vector index. A Basic partition holds 15 GB of storage and 5 GB of vectors, so storage runs out first, at a little over 800 documents of this size.
If you prefer the portal: the Import data wizard (scenario "RAG") creates the same four objects. According to the docs, it always chunks with 2,000 characters and 500 characters of overlap, and that setting cannot be changed. That matters, as the test below shows.
Step 2: hybrid search and an answer with citations
The query code fits into one module. retrieve sends a hybrid query: the
question as search text for BM25, and a VectorizableTextQuery that the
search service embeds itself. With query_type="semantic" the semantic
ranker reorders the results. answer numbers the top five chunks and tells
gpt-5.4-mini to cite them.
k_nearest_neighbors=50 gives the semantic ranker the full 50 candidates it
can rerank. The OpenAI client uses the v1 endpoint of the Foundry resource
and a token from Entra ID instead of an API key; model is the deployment
name. For the question "From when do the obligations for general-purpose AI
models apply?", the pipeline answered:
Source [1] is the chunk with Recital 179, which gives exactly that date.
Your application maps the numbers back to the chunks (title, chunk_id)
and can show them as links.
Test results: five questions on the EU AI Act
We asked five questions whose answers are clearly stated in the text. For each question we checked whether the chunk with the answer was among the top five results of each search mode, and whether the final answer was correct. One question was in German against the English document.
| Question | Keyword | Vector | Hybrid | Hybrid + semantic | Answer |
|---|---|---|---|---|---|
| Maximum fine for a prohibited AI practice | not found | rank 5 | not found | rank 1 | correct: EUR 35 million or 7 % (Art. 99(3)) |
| What chatbot providers must tell users | not found | rank 1 | rank 4 | rank 1 | correct (Art. 50(1)) |
| Application date of the GPAI obligations | not found | not found | not found | rank 1 | correct: 2 August 2025 |
| Deployer duties for human oversight (German) | not found | rank 3 | rank 3 | rank 1 | correct, answered in German (Art. 26(2)) |
| Is CV screening AI high-risk? | rank 1 | rank 1 | rank 1 | rank 1 | correct (Annex III) |
Hybrid search with the semantic ranker put the right chunk in first place for every question. Keyword search alone found only one of them, and the German question got no keyword match at all because the document is in English. Vector search handled the cross-language case, and the reranker made the difference for the fines and date questions.
Query latency (median and 90th percentile over 15 queries per mode, client in Austria, service in France Central):
| Mode | Median | p90 |
|---|---|---|
| Keyword | 40 ms | 68 ms |
| Vector | 108 ms | 143 ms |
| Hybrid | 217 ms | 263 ms |
| Hybrid + semantic | 269 ms | 340 ms |
The complete answer including the model call took between 0.9 and 4.2
seconds, with a median of 1.2 seconds. Per question, gpt-5.4-mini read
about 1,050 input tokens and wrote 50 to 270 output tokens.
What the first two attempts got wrong
The table shows the final configuration. We got there in three runs:
| Setup | Correct answers |
|---|---|
| One 144-page PDF, 2,000/500 characters per chunk | 3 of 5 (fines: "not in the sources"; GPAI date: wrong) |
| Three PDF parts, 2,000/500 characters | 4 of 5 (GPAI date: wrong) |
| Three PDF parts, 1,000/200 characters | 5 of 5 |
In the first run, the indexer reported success with a single warning:
Truncated extracted text to '524288' characters. The index ended in the
middle of Article 99. Everything after that was missing: the fines, Article
113 with the application dates and all annexes. The model said correctly
that the sources did not contain the fine, and it answered the high-risk
question from the recitals instead of Annex III.
The second fix concerns the chunk size. With 2,000-character chunks, the passage on the application dates sat in a chunk that also contained Article 1, and Article 113 itself only says "Chapter V" without mentioning general-purpose AI. Neither chunk made it into the top 50. With 1,000 characters, the Recital 179 chunk was in first place. Smaller chunks are not always better, but for legal texts and other documents with many short, self-contained provisions they helped clearly here. Test this with your own documents and questions before you settle on the wizard default.
Knowledge bases and agentic retrieval
A knowledge base sits on top of an existing index. In the GA version
(2026-04-01) it does not plan queries: you pass one or more search intents,
the service runs them in parallel with semantic reranking, and returns the
merged chunks with references. The SDK 12.0.0 supports this:
The response contains the chunks as a JSON array (ref_id, title,
content), ready to be passed to a model. In our run the Recital 179 chunk
came first. The GA version does about the same as the classic hybrid query,
but it adds two things: it can query several knowledge sources (indexes,
blob, OneLake, web) in one call, and agents in Foundry can use it directly
as a tool.
The more interesting part is still in preview. With
2026-08-01-preview, a model reference and reasoning effort low, we sent
the original date question to a knowledge base on the index with 2,000-character
chunks, the one where classic RAG had failed. gpt-5.4-mini turned the
question into three subqueries, and the Recital 179 passage was in the
result. The call took 2.0 seconds: 0.86 seconds for query planning and about
0.3 seconds per subquery, run in parallel. Query planning compensates for a
less than ideal chunk size, at the cost of one extra model call per
question. Since it is a preview feature, keep it out of production for now.
Pitfalls
Large files are cut off silently
Each tier extracts only a limited number of characters per file: 256,000 on Free, 512,000 on Basic, 4 million on S1 (see the limits page). The indexer still reports "success" and leaves a warning in the run history. Check the warnings after every run, split large files, or use S1. The maximum file size is 16 MB on Free and Basic.
The wizard's chunk size is fixed
The portal wizard always chunks with 2,000/500 characters. If you need different values, define the skillset in code as above, or edit it after the wizard has run.
Popular EU regions are closed to new services
In four of the most common European regions you currently cannot create a new search service. Check the region list before planning, and keep in mind that search service and model deployments can sit in different regions.
The free semantic ranker plan runs out
New services use the free plan with a monthly
allowance; the pricing page
lists 1,000 free requests. After that, queries return a billing error
until you switch the service to the standard plan (az search service update --semantic-search standard). Note that the pricing page and the semantic
ranker docs disagree on whether the Free tier gets the ranker at all. Plan
for Basic.
DefaultAzureCredential picks the wrong tenant
DefaultAzureCredential uses the Azure CLI login. If the
CLI's default subscription belongs to another tenant, every request fails
with an authentication error. az account set --subscription <id> fixes it.
Chunks carry no page numbers
The default text extraction does not record which page
a chunk comes from. For the AI Act, article numbers in the text were enough
for citations. If you need page numbers or bounding boxes, use the
Document Layout
skill
with locationMetadata. It is billed through Document Intelligence beyond
20 documents per indexer per day.
Index statistics lag behind
Right after indexing, the index statistics showed
0 documents and 0 bytes for several minutes while queries already returned
results. Use get_document_count() to check what is searchable.
What Azure AI Search costs
Prices from the Azure Retail Prices API for France Central, retrieved on 6 November 2026. Monthly values assume 730 hours with one replica and one partition.
| Tier | Price per hour | Per month | Storage per partition | Vector index per partition |
|---|---|---|---|---|
| Free | 0 | 0 | 50 MB total | n/a |
| Basic | 0.0907 EUR | about 66 EUR | 15 GB | 5 GB |
| Standard S1 | 0.3016 EUR | about 220 EUR | 160 GB | 35 GB |
| Standard S2 | 1.2065 EUR | about 881 EUR | 512 GB | 150 GB |
On top of that come usage-based costs:
| Item | Price |
|---|---|
| Semantic ranker | 1,000 requests per month free, then 0.8977 EUR per 1,000 |
| Agentic retrieval | 50 million tokens per month free, then 0.022 USD per million tokens |
| gpt-5.4-mini, DataZoneStandard | 0.825 USD per million input tokens, 4.95 USD per million output tokens |
| text-embedding-3-small, DataZoneStandard | 0.022 USD per million tokens |
At about 1,050 input and 120 output tokens per answer, one answer costs about 0.0015 USD for the model. Embedding the entire AI Act once costs well under one cent. In practice, the monthly bill is mostly the search tier: for a few thousand questions a month on Basic you pay about 66 EUR plus a few euros for the model and the semantic ranker. Every additional replica or partition adds the same hourly price again, so a Basic service with two replicas costs about 132 EUR. Our complete test run cost less than one euro.
Microsoft also offers a Serverless tier as a preview, billed by compute units per hour and stored GB, with scale to zero and 1 GB per index. It is only available in some regions, has no SLA and cannot be migrated to the dedicated tiers. For a proof of concept with little traffic it may become the cheaper option.
Our assessment
For companies that already run on Azure, Azure AI Search is the obvious place to start with RAG. The integrated pipeline saves you from writing chunking, embedding and sync code, the indexer picks up changed files on its own, and hybrid search with the semantic ranker gave good retrieval in our test without any tuning beyond the chunk size. With managed identities and Entra ID roles, no keys appear in code, and with France Central or Switzerland North plus DataZoneStandard deployments, data stays in the EU or in Switzerland.
The weak spots are easy to overlook. The per-file extraction limit is easy to miss, the wizard's chunk size cannot be changed, and the most popular EU regions are currently closed to new services. Agentic retrieval with query planning is a real improvement, but most of it is still in preview. Before going live, test retrieval with your own documents and a set of real questions, as we did here. Our post on evaluating RAG with Ragas shows how to automate that. If the reranking quality is not enough, a late-interaction reranker such as ColBERT is an alternative to the semantic ranker.
If your knowledge lives in SharePoint and Microsoft 365, also look at SharePoint lists as knowledge sources in Copilot Studio. For agents that use the index as a tool, see our guide to the Microsoft Agent Framework.
We build RAG systems on Azure for companies in Austria, Germany and Switzerland, from document preparation to evaluation. If you are planning one, our AI consulting is a good place to start, or get in touch directly.
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms



