Azure AI Search RAG Tutorial: Build RAG on Azure with Python

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 16 min read
Azure AI Search RAG Tutorial: Build RAG on Azure with Python
On this page

Azure AI Search is Microsoft's managed search service, and on Azure it is the standard building block for the retrieval half of RAG (Retrieval-Augmented Generation). It extracts text from your files, splits it into chunks, embeds them with an Azure OpenAI model, and answers queries with a combination of keyword search, vector search and a reranker. A chat model in Microsoft Foundry then writes the answer from the chunks it gets back.

This tutorial builds that pipeline end to end with the Python SDK (azure-search-documents 12.0.0) and tests it on a real document: the 144-page EU AI Act. The final setup answered all five test questions correctly, with article references, in about 1.2 seconds per answer. It took two fixes to get there, and both are problems you will run into with your own documents: the Basic tier quietly cut off the last 15 % of the PDF, and the default chunk size missed a question about application dates. We tested on 6 November 2026; prices and region availability are from the same day.

How RAG with Azure AI Search works

RAG has two phases. At indexing time, documents are split into chunks and each chunk is stored with an embedding. At query time, the system finds the chunks that best match the question and passes them to a language model as context. Azure AI Search covers the first phase completely and the search part of the second.

RAG pipeline with Azure AI Search: indexer and skillset fill the index, a hybrid query with semantic ranker feeds gpt-5.4-mini

These are the objects you create in Azure AI Search:

ObjectWhat it does
Data sourceConnection to Blob Storage, SharePoint, OneLake, Azure SQL, Cosmos DB and others
SkillsetProcessing steps during indexing, here the Text Split skill (chunking) and the Azure OpenAI Embedding skill
IndexThe searchable store: text fields for keyword search, vector fields for similarity search
Index projectionWrites each chunk as its own search document instead of one document per file
IndexerRuns the pipeline on demand or on a schedule and only processes new or changed files
VectorizerEmbeds the user's question at query time with the same model, so your code sends plain text

Microsoft calls the combination of skillset, index projection and vectorizer "integrated vectorization". Your application never computes an embedding itself.

At query time, a hybrid query runs a BM25 keyword search and a vector search (HNSW) in parallel and merges both result lists with Reciprocal Rank Fusion (RRF). The semantic ranker then reranks the top 50 results with a multilingual model derived from Bing. The top five chunks go to the chat model.

Classic RAG, agentic retrieval and Foundry IQ

Microsoft's RAG overview now describes two approaches. "Classic RAG" is the pipeline above: your code sends one hybrid query and calls the model. "Agentic retrieval" lets an LLM plan several subqueries, runs them in parallel and returns merged results. Its resource is called a knowledge base, and Foundry IQ is the name for knowledge bases used by agents in the Microsoft Foundry portal.

The status is mixed. Knowledge bases with minimal, extractive retrieval have been generally available since REST API 2026-04-01. LLM query planning, answer synthesis and the reasoning effort levels low and medium are still in preview (2026-08-01-preview), and the portal shows all agentic retrieval features in preview mode only. We test both further down.

If you used "Azure OpenAI On Your Data" (the data_sources parameter in the chat completions API, or "Add your data" in the playground): Microsoft retired it on 14 October 2026. Existing search indexes keep working. You now call the search service yourself, as shown below, or move to a knowledge base.

Choose the tier and region

For data residency in the DACH region, the region matters as much as the tier. The region list (last updated 3 November 2026) marks Sweden Central, West Europe, Germany West Central and North Europe as "high demand", which "prevents the creation of new search services". Existing services there keep running. France Central and Switzerland North accept new services, and all six regions support the semantic ranker and agentic retrieval.

The chat and embedding models can run in a different region than the search service. We checked model availability with az cognitiveservices model list on 6 November 2026:

RegionSearch servicegpt-5.4-minitext-embedding-3-small
France Centralnew services possibleDataZoneStandard, GlobalStandardDataZoneStandard, GlobalStandard
Switzerland Northnew services possibleDataZoneStandard, GlobalStandardStandard (regional), GlobalStandard
Germany West Centralhigh demand, no new servicesDataZoneStandard, GlobalStandardDataZoneStandard, GlobalStandard
Sweden Centralhigh demand, no new servicesDataZoneStandard, GlobalStandardDataZoneStandard, GlobalStandard

We used France Central for everything and deployed both models as DataZoneStandard. According to the deployment types page, the service then processes data only within the EU Data Boundary (which can include EFTA countries such as Switzerland and Norway), and data at rest stays in the resource's geography. A GlobalStandard deployment is about 9 % cheaper but "may be processed in any Azure region".

For the tier, Free is fine for a first look, but each subscription gets only one free service, with 50 MB of storage, three indexes and indexer runs capped at three minutes. Basic is the smallest tier for real documents. The import quickstart also requires Basic or higher for managed identities, which we use for every connection.

Set up the Azure resources

You need an Azure subscription, the Azure CLI and Python 3.10 or later. These are the commands we ran; replace the placeholders with your own values:

1SUB="<subscription-id>"
2RG="rg-rag-demo"
3LOCATION="francecentral"
4SUFFIX="<unique-suffix>"
5
6az group create --subscription $SUB -n $RG -l $LOCATION
7
8az search service create --subscription $SUB -g $RG -n srch-rag-$SUFFIX -l $LOCATION \
9 --sku basic --partition-count 1 --replica-count 1 \
10 --auth-options aadOrApiKey --aad-auth-failure-mode http401WithBearerChallenge \
11 --identity-type SystemAssigned --semantic-search free
12
13az cognitiveservices account create --subscription $SUB -g $RG -n aif-rag-$SUFFIX -l $LOCATION \
14 --kind AIServices --sku S0 --custom-domain aif-rag-$SUFFIX --yes
15
16az storage account create --subscription $SUB -g $RG -n strag$SUFFIX -l $LOCATION \
17 --sku Standard_LRS --kind StorageV2 --allow-blob-public-access false --min-tls-version TLS1_2

--auth-options aadOrApiKey enables Microsoft Entra ID (RBAC) next to the API keys, --identity-type SystemAssigned gives the search service its own managed identity, and --semantic-search free activates the free plan of the semantic ranker. The AIServices account is a Microsoft Foundry resource; it needs a custom subdomain so Azure AI Search can call it. Next, deploy the two models:

1az cognitiveservices account deployment create --subscription $SUB -g $RG -n aif-rag-$SUFFIX \
2 --deployment-name gpt-5.4-mini --model-name gpt-5.4-mini --model-version 2026-03-17 \
3 --model-format OpenAI --sku-name DataZoneStandard --sku-capacity 100
4
5az cognitiveservices account deployment create --subscription $SUB -g $RG -n aif-rag-$SUFFIX \
6 --deployment-name text-embedding-3-small --model-name text-embedding-3-small --model-version 1 \
7 --model-format OpenAI --sku-name DataZoneStandard --sku-capacity 150

We picked gpt-5.4-mini because Azure already lists gpt-4.1-mini as "Legacy" (retirement 14 April 2027). The capacity is in thousands of tokens per minute.

Then assign the roles. The search service's managed identity reads the blobs and calls the embedding model; your own user account manages the index and calls the chat model. No keys end up in code or in the index definition.

1SRCH_ID=$(az search service show --subscription $SUB -g $RG -n srch-rag-$SUFFIX --query id -o tsv)
2SRCH_MI=$(az search service show --subscription $SUB -g $RG -n srch-rag-$SUFFIX --query identity.principalId -o tsv)
3AIF_ID=$(az cognitiveservices account show --subscription $SUB -g $RG -n aif-rag-$SUFFIX --query id -o tsv)
4ST_ID=$(az storage account show --subscription $SUB -g $RG -n strag$SUFFIX --query id -o tsv)
5ME="<your Entra object ID>"
6
7az role assignment create --subscription $SUB --assignee-object-id $SRCH_MI --assignee-principal-type ServicePrincipal --role "Cognitive Services OpenAI User" --scope $AIF_ID
8az role assignment create --subscription $SUB --assignee-object-id $SRCH_MI --assignee-principal-type ServicePrincipal --role "Storage Blob Data Reader" --scope $ST_ID
9az role assignment create --subscription $SUB --assignee-object-id $ME --assignee-principal-type User --role "Search Service Contributor" --scope $SRCH_ID
10az role assignment create --subscription $SUB --assignee-object-id $ME --assignee-principal-type User --role "Search Index Data Contributor" --scope $SRCH_ID
11az role assignment create --subscription $SUB --assignee-object-id $ME --assignee-principal-type User --role "Cognitive Services OpenAI User" --scope $AIF_ID
12az role assignment create --subscription $SUB --assignee-object-id $ME --assignee-principal-type User --role "Storage Blob Data Contributor" --scope $ST_ID

Upload the document

Our test document is the EU AI Act, Regulation (EU) 2024/1689 as an English PDF from EUR-Lex (144 pages, 2.6 MB). The EU allows reuse of EUR-Lex documents with attribution. Its long articles, cross-references and annexes make it a good stand-in for contracts, standards and manuals.

The Basic tier only extracts the first 512,000 characters of each file (more on that under pitfalls), so we split the PDF into three parts of 48 pages each before uploading:

1qpdf ai-act-en.pdf --pages . 1-48 -- ai-act-en-part1.pdf
2qpdf ai-act-en.pdf --pages . 49-96 -- ai-act-en-part2.pdf
3qpdf ai-act-en.pdf --pages . 97-144 -- ai-act-en-part3.pdf
4
5az storage container create --subscription $SUB --account-name strag$SUFFIX -n docs --auth-mode login
6az storage blob upload-batch --subscription $SUB --account-name strag$SUFFIX -d docs -s ./parts --auth-mode login

Install the Python packages

1uv add azure-search-documents==12.0.0 azure-identity openai

We used azure-search-documents 12.0.0 (released 1 May 2026, targets REST API 2026-04-01), azure-identity 1.27.0 and openai 3.46.0. The scripts read three environment variables:

1export AZURE_SEARCH_ENDPOINT=https://srch-rag-<suffix>.search.windows.net
2export AZURE_OPENAI_ENDPOINT=https://aif-rag-<suffix>.openai.azure.com
3export STORAGE_RESOURCE_ID=$ST_ID

DefaultAzureCredential signs in with your Azure CLI login. If your account can see several tenants, run az account set --subscription <id> first: the CLI credential requests tokens for the tenant of the default subscription, and a token from the wrong tenant fails with an authentication error.

Step 1: create the index and the indexing pipeline

This script creates the data source, the index, the skillset and the indexer. The indexer starts as soon as it is created; the script waits until the run has finished.

1import os
2import time
3
4from azure.identity import DefaultAzureCredential
5from azure.search.documents.indexes import SearchIndexClient, SearchIndexerClient
6from azure.search.documents.indexes.models import (
7 AzureOpenAIEmbeddingSkill,
8 AzureOpenAIVectorizer,
9 AzureOpenAIVectorizerParameters,
10 HnswAlgorithmConfiguration,
11 IndexProjectionMode,
12 InputFieldMappingEntry,
13 OutputFieldMappingEntry,
14 SearchField,
15 SearchFieldDataType,
16 SearchIndex,
17 SearchIndexer,
18 SearchIndexerDataContainer,
19 SearchIndexerDataSourceConnection,
20 SearchIndexerIndexProjection,
21 SearchIndexerIndexProjectionSelector,
22 SearchIndexerIndexProjectionsParameters,
23 SearchIndexerSkillset,
24 SemanticConfiguration,
25 SemanticField,
26 SemanticPrioritizedFields,
27 SemanticSearch,
28 SplitSkill,
29 VectorSearch,
30 VectorSearchProfile,
31)
32
33search_endpoint = os.environ["AZURE_SEARCH_ENDPOINT"] # https://<service>.search.windows.net
34openai_endpoint = os.environ["AZURE_OPENAI_ENDPOINT"] # https://<resource>.openai.azure.com
35storage_id = os.environ["STORAGE_RESOURCE_ID"] # /subscriptions/.../storageAccounts/<name>
36name = "ai-act"
37
38credential = DefaultAzureCredential()
39index_client = SearchIndexClient(search_endpoint, credential)
40indexer_client = SearchIndexerClient(search_endpoint, credential)
41
42# 1. Data source: a blob container, accessed with the search service's managed identity
43data_source = SearchIndexerDataSourceConnection(
44 name=f"{name}-blob",
45 type="azureblob",
46 connection_string=f"ResourceId={storage_id};",
47 container=SearchIndexerDataContainer(name="docs"),
48)
49indexer_client.create_or_update_data_source_connection(data_source)
50
51# 2. Index: one search document per chunk, with text and vector fields
52embedding_params = AzureOpenAIVectorizerParameters(
53 resource_url=openai_endpoint,
54 deployment_name="text-embedding-3-small",
55 model_name="text-embedding-3-small",
56)
57index = SearchIndex(
58 name=name,
59 fields=[
60 SearchField(name="chunk_id", type=SearchFieldDataType.STRING, key=True,
61 searchable=True, analyzer_name="keyword", sortable=True),
62 SearchField(name="parent_id", type=SearchFieldDataType.STRING, filterable=True),
63 SearchField(name="title", type=SearchFieldDataType.STRING, searchable=True),
64 SearchField(name="chunk", type=SearchFieldDataType.STRING, searchable=True,
65 analyzer_name="en.microsoft"),
66 SearchField(name="text_vector", type="Collection(Edm.Single)", searchable=True,
67 vector_search_dimensions=1536, vector_search_profile_name="hnsw-profile"),
68 ],
69 vector_search=VectorSearch(
70 algorithms=[HnswAlgorithmConfiguration(name="hnsw")],
71 vectorizers=[AzureOpenAIVectorizer(vectorizer_name="openai", parameters=embedding_params)],
72 profiles=[VectorSearchProfile(name="hnsw-profile", algorithm_configuration_name="hnsw",
73 vectorizer_name="openai")],
74 ),
75 semantic_search=SemanticSearch(
76 default_configuration_name="default",
77 configurations=[SemanticConfiguration(
78 name="default",
79 prioritized_fields=SemanticPrioritizedFields(
80 title_field=SemanticField(field_name="title"),
81 content_fields=[SemanticField(field_name="chunk")],
82 ),
83 )],
84 ),
85)
86index_client.create_or_update_index(index)
87
88# 3. Skillset: split the extracted text into chunks, embed every chunk
89split = SplitSkill(
90 name="split",
91 context="/document",
92 text_split_mode="pages",
93 maximum_page_length=1000,
94 page_overlap_length=200,
95 inputs=[InputFieldMappingEntry(name="text", source="/document/content")],
96 outputs=[OutputFieldMappingEntry(name="textItems", target_name="pages")],
97)
98embed = AzureOpenAIEmbeddingSkill(
99 name="embed",
100 context="/document/pages/*",
101 resource_url=openai_endpoint,
102 deployment_name="text-embedding-3-small",
103 model_name="text-embedding-3-small",
104 dimensions=1536,
105 inputs=[InputFieldMappingEntry(name="text", source="/document/pages/*")],
106 outputs=[OutputFieldMappingEntry(name="embedding", target_name="text_vector")],
107)
108skillset = SearchIndexerSkillset(
109 name=f"{name}-skillset",
110 skills=[split, embed],
111 # index projections write each chunk as its own search document
112 index_projection=SearchIndexerIndexProjection(
113 selectors=[SearchIndexerIndexProjectionSelector(
114 target_index_name=name,
115 parent_key_field_name="parent_id",
116 source_context="/document/pages/*",
117 mappings=[
118 InputFieldMappingEntry(name="chunk", source="/document/pages/*"),
119 InputFieldMappingEntry(name="text_vector", source="/document/pages/*/text_vector"),
120 InputFieldMappingEntry(name="title", source="/document/metadata_storage_name"),
121 ],
122 )],
123 parameters=SearchIndexerIndexProjectionsParameters(
124 projection_mode=IndexProjectionMode.SKIP_INDEXING_PARENT_DOCUMENTS,
125 ),
126 ),
127)
128indexer_client.create_or_update_skillset(skillset)
129
130# 4. Indexer: connects data source, skillset and index, runs on creation
131indexer = SearchIndexer(
132 name=f"{name}-indexer",
133 data_source_name=data_source.name,
134 skillset_name=skillset.name,
135 target_index_name=name,
136)
137indexer_client.create_or_update_indexer(indexer)
138
139start = time.time()
140while True:
141 time.sleep(10)
142 status = indexer_client.get_indexer_status(indexer.name)
143 run = status.last_result
144 if run and run.status not in ("inProgress", "reset"):
145 break
146print(f"status={run.status} items={run.item_count} failed={run.failed_item_count} "
147 f"errors={[e.error_message for e in run.errors]} took~{time.time() - start:.0f}s")

A few details are easy to get wrong:

  • The key field chunk_id needs the keyword analyzer, otherwise index projections cannot write the generated chunk IDs.
  • The vectorizer in the index and the embedding skill must point to the same model with the same dimensions. Otherwise queries and chunks end up in different vector spaces.
  • en.microsoft is the language analyzer for keyword search. For German documents use de.microsoft.
  • No connection string contains a key. The data source uses ResourceId=... and both the skill and the vectorizer authenticate with the search service's managed identity.

The run on the three parts took 73 seconds and produced 809 chunks. The index uses 18.7 MB, 5.0 MB of which is the vector index. A Basic partition holds 15 GB of storage and 5 GB of vectors, so storage runs out first, at a little over 800 documents of this size.

If you prefer the portal: the Import data wizard (scenario "RAG") creates the same four objects. According to the docs, it always chunks with 2,000 characters and 500 characters of overlap, and that setting cannot be changed. That matters, as the test below shows.

Step 2: hybrid search and an answer with citations

The query code fits into one module. retrieve sends a hybrid query: the question as search text for BM25, and a VectorizableTextQuery that the search service embeds itself. With query_type="semantic" the semantic ranker reorders the results. answer numbers the top five chunks and tells gpt-5.4-mini to cite them.

1import os
2
3from azure.identity import DefaultAzureCredential, get_bearer_token_provider
4from azure.search.documents import SearchClient
5from azure.search.documents.models import VectorizableTextQuery
6from openai import OpenAI
7
8credential = DefaultAzureCredential()
9search = SearchClient(os.environ["AZURE_SEARCH_ENDPOINT"], "ai-act", credential)
10llm = OpenAI(
11 base_url=f"{os.environ['AZURE_OPENAI_ENDPOINT']}/openai/v1/",
12 api_key=get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default"),
13)
14
15
16def retrieve(question: str, mode: str = "hybrid-semantic", top: int = 5) -> list[dict]:
17 vector = VectorizableTextQuery(text=question, k_nearest_neighbors=50, fields="text_vector")
18 args = {"select": ["chunk_id", "title", "chunk"], "top": top}
19 if mode in ("keyword", "hybrid", "hybrid-semantic"):
20 args["search_text"] = question
21 if mode in ("vector", "hybrid", "hybrid-semantic"):
22 args["vector_queries"] = [vector]
23 if mode == "hybrid-semantic":
24 args["query_type"] = "semantic"
25 args["semantic_configuration_name"] = "default"
26 return list(search.search(**args))
27
28
29SYSTEM = """You answer questions about the EU AI Act using only the numbered sources.
30Cite every statement with its source number in square brackets, e.g. [2], and name the
31Article or Annex when the source shows it. If the sources do not contain the answer,
32say so instead of guessing. Answer in the language of the question."""
33
34
35def answer(question: str) -> tuple[str, list[dict]]:
36 chunks = retrieve(question)
37 sources = "\n\n".join(f"[{i}] {c['chunk']}" for i, c in enumerate(chunks, start=1))
38 response = llm.responses.create(
39 model="gpt-5.4-mini", # deployment name
40 instructions=SYSTEM,
41 input=f"Sources:\n{sources}\n\nQuestion: {question}",
42 )
43 return response.output_text, chunks
44
45
46if __name__ == "__main__":
47 import sys
48
49 text, chunks = answer(sys.argv[1])
50 print(text)
51 for i, c in enumerate(chunks, start=1):
52 print(f"[{i}] {c['title']} {c['chunk'][:80]!r}")

k_nearest_neighbors=50 gives the semantic ranker the full 50 candidates it can rerank. The OpenAI client uses the v1 endpoint of the Foundry resource and a token from Entra ID instead of an API key; model is the deployment name. For the question "From when do the obligations for general-purpose AI models apply?", the pipeline answered:

1The obligations for providers of general-purpose AI models apply from **2 August 2025**. [1]
2
3The sources also say these obligations apply **once the general-purpose AI models are placed on the market**. [2]

Source [1] is the chunk with Recital 179, which gives exactly that date. Your application maps the numbers back to the chunks (title, chunk_id) and can show them as links.

Test results: five questions on the EU AI Act

We asked five questions whose answers are clearly stated in the text. For each question we checked whether the chunk with the answer was among the top five results of each search mode, and whether the final answer was correct. One question was in German against the English document.

QuestionKeywordVectorHybridHybrid + semanticAnswer
Maximum fine for a prohibited AI practicenot foundrank 5not foundrank 1correct: EUR 35 million or 7 % (Art. 99(3))
What chatbot providers must tell usersnot foundrank 1rank 4rank 1correct (Art. 50(1))
Application date of the GPAI obligationsnot foundnot foundnot foundrank 1correct: 2 August 2025
Deployer duties for human oversight (German)not foundrank 3rank 3rank 1correct, answered in German (Art. 26(2))
Is CV screening AI high-risk?rank 1rank 1rank 1rank 1correct (Annex III)

Hybrid search with the semantic ranker put the right chunk in first place for every question. Keyword search alone found only one of them, and the German question got no keyword match at all because the document is in English. Vector search handled the cross-language case, and the reranker made the difference for the fines and date questions.

Query latency (median and 90th percentile over 15 queries per mode, client in Austria, service in France Central):

ModeMedianp90
Keyword40 ms68 ms
Vector108 ms143 ms
Hybrid217 ms263 ms
Hybrid + semantic269 ms340 ms

The complete answer including the model call took between 0.9 and 4.2 seconds, with a median of 1.2 seconds. Per question, gpt-5.4-mini read about 1,050 input tokens and wrote 50 to 270 output tokens.

What the first two attempts got wrong

The table shows the final configuration. We got there in three runs:

SetupCorrect answers
One 144-page PDF, 2,000/500 characters per chunk3 of 5 (fines: "not in the sources"; GPAI date: wrong)
Three PDF parts, 2,000/500 characters4 of 5 (GPAI date: wrong)
Three PDF parts, 1,000/200 characters5 of 5

In the first run, the indexer reported success with a single warning: Truncated extracted text to '524288' characters. The index ended in the middle of Article 99. Everything after that was missing: the fines, Article 113 with the application dates and all annexes. The model said correctly that the sources did not contain the fine, and it answered the high-risk question from the recitals instead of Annex III.

The second fix concerns the chunk size. With 2,000-character chunks, the passage on the application dates sat in a chunk that also contained Article 1, and Article 113 itself only says "Chapter V" without mentioning general-purpose AI. Neither chunk made it into the top 50. With 1,000 characters, the Recital 179 chunk was in first place. Smaller chunks are not always better, but for legal texts and other documents with many short, self-contained provisions they helped clearly here. Test this with your own documents and questions before you settle on the wizard default.

Knowledge bases and agentic retrieval

A knowledge base sits on top of an existing index. In the GA version (2026-04-01) it does not plan queries: you pass one or more search intents, the service runs them in parallel with semantic reranking, and returns the merged chunks with references. The SDK 12.0.0 supports this:

1import os
2
3from azure.identity import DefaultAzureCredential
4from azure.search.documents.indexes import SearchIndexClient
5from azure.search.documents.indexes.models import (
6 KnowledgeBase,
7 KnowledgeSourceReference,
8 SearchIndexKnowledgeSource,
9 SearchIndexKnowledgeSourceParameters,
10)
11from azure.search.documents.knowledgebases import KnowledgeBaseRetrievalClient
12from azure.search.documents.knowledgebases.models import (
13 KnowledgeBaseRetrievalRequest,
14 KnowledgeRetrievalSemanticIntent,
15 SearchIndexKnowledgeSourceParams,
16)
17
18endpoint = os.environ["AZURE_SEARCH_ENDPOINT"]
19credential = DefaultAzureCredential()
20index_client = SearchIndexClient(endpoint, credential)
21
22# A knowledge source wraps the existing index, a knowledge base groups knowledge sources
23index_client.create_or_update_knowledge_source(SearchIndexKnowledgeSource(
24 name="ai-act-source",
25 search_index_parameters=SearchIndexKnowledgeSourceParameters(search_index_name="ai-act"),
26))
27index_client.create_or_update_knowledge_base(KnowledgeBase(
28 name="ai-act-kb",
29 knowledge_sources=[KnowledgeSourceReference(name="ai-act-source")],
30))
31
32kb = KnowledgeBaseRetrievalClient(endpoint, credential, knowledge_base_name="ai-act-kb")
33result = kb.retrieve(KnowledgeBaseRetrievalRequest(
34 # GA (2026-04-01): you pass the queries yourself, no LLM query planning
35 intents=[
36 KnowledgeRetrievalSemanticIntent(search="maximum fine for prohibited AI practices"),
37 KnowledgeRetrievalSemanticIntent(search="date of application of general-purpose AI obligations"),
38 ],
39 knowledge_source_params=[SearchIndexKnowledgeSourceParams(
40 knowledge_source_name="ai-act-source", include_references=True,
41 )],
42))
43print(result.response[0].content[0].text[:600])
44for ref in result.references[:6]:
45 print(ref.doc_key[-12:], round(ref.reranker_score or 0, 2))

The response contains the chunks as a JSON array (ref_id, title, content), ready to be passed to a model. In our run the Recital 179 chunk came first. The GA version does about the same as the classic hybrid query, but it adds two things: it can query several knowledge sources (indexes, blob, OneLake, web) in one call, and agents in Foundry can use it directly as a tool.

The more interesting part is still in preview. With 2026-08-01-preview, a model reference and reasoning effort low, we sent the original date question to a knowledge base on the index with 2,000-character chunks, the one where classic RAG had failed. gpt-5.4-mini turned the question into three subqueries, and the Recital 179 passage was in the result. The call took 2.0 seconds: 0.86 seconds for query planning and about 0.3 seconds per subquery, run in parallel. Query planning compensates for a less than ideal chunk size, at the cost of one extra model call per question. Since it is a preview feature, keep it out of production for now.

Pitfalls

Large files are cut off silently

Each tier extracts only a limited number of characters per file: 256,000 on Free, 512,000 on Basic, 4 million on S1 (see the limits page). The indexer still reports "success" and leaves a warning in the run history. Check the warnings after every run, split large files, or use S1. The maximum file size is 16 MB on Free and Basic.

The wizard's chunk size is fixed

The portal wizard always chunks with 2,000/500 characters. If you need different values, define the skillset in code as above, or edit it after the wizard has run.

In four of the most common European regions you currently cannot create a new search service. Check the region list before planning, and keep in mind that search service and model deployments can sit in different regions.

The free semantic ranker plan runs out

New services use the free plan with a monthly allowance; the pricing page lists 1,000 free requests. After that, queries return a billing error until you switch the service to the standard plan (az search service update --semantic-search standard). Note that the pricing page and the semantic ranker docs disagree on whether the Free tier gets the ranker at all. Plan for Basic.

DefaultAzureCredential picks the wrong tenant

DefaultAzureCredential uses the Azure CLI login. If the CLI's default subscription belongs to another tenant, every request fails with an authentication error. az account set --subscription <id> fixes it.

Chunks carry no page numbers

The default text extraction does not record which page a chunk comes from. For the AI Act, article numbers in the text were enough for citations. If you need page numbers or bounding boxes, use the Document Layout skill with locationMetadata. It is billed through Document Intelligence beyond 20 documents per indexer per day.

Index statistics lag behind

Right after indexing, the index statistics showed 0 documents and 0 bytes for several minutes while queries already returned results. Use get_document_count() to check what is searchable.

What Azure AI Search costs

Prices from the Azure Retail Prices API for France Central, retrieved on 6 November 2026. Monthly values assume 730 hours with one replica and one partition.

TierPrice per hourPer monthStorage per partitionVector index per partition
Free0050 MB totaln/a
Basic0.0907 EURabout 66 EUR15 GB5 GB
Standard S10.3016 EURabout 220 EUR160 GB35 GB
Standard S21.2065 EURabout 881 EUR512 GB150 GB

On top of that come usage-based costs:

ItemPrice
Semantic ranker1,000 requests per month free, then 0.8977 EUR per 1,000
Agentic retrieval50 million tokens per month free, then 0.022 USD per million tokens
gpt-5.4-mini, DataZoneStandard0.825 USD per million input tokens, 4.95 USD per million output tokens
text-embedding-3-small, DataZoneStandard0.022 USD per million tokens

At about 1,050 input and 120 output tokens per answer, one answer costs about 0.0015 USD for the model. Embedding the entire AI Act once costs well under one cent. In practice, the monthly bill is mostly the search tier: for a few thousand questions a month on Basic you pay about 66 EUR plus a few euros for the model and the semantic ranker. Every additional replica or partition adds the same hourly price again, so a Basic service with two replicas costs about 132 EUR. Our complete test run cost less than one euro.

Microsoft also offers a Serverless tier as a preview, billed by compute units per hour and stored GB, with scale to zero and 1 GB per index. It is only available in some regions, has no SLA and cannot be migrated to the dedicated tiers. For a proof of concept with little traffic it may become the cheaper option.

Our assessment

For companies that already run on Azure, Azure AI Search is the obvious place to start with RAG. The integrated pipeline saves you from writing chunking, embedding and sync code, the indexer picks up changed files on its own, and hybrid search with the semantic ranker gave good retrieval in our test without any tuning beyond the chunk size. With managed identities and Entra ID roles, no keys appear in code, and with France Central or Switzerland North plus DataZoneStandard deployments, data stays in the EU or in Switzerland.

The weak spots are easy to overlook. The per-file extraction limit is easy to miss, the wizard's chunk size cannot be changed, and the most popular EU regions are currently closed to new services. Agentic retrieval with query planning is a real improvement, but most of it is still in preview. Before going live, test retrieval with your own documents and a set of real questions, as we did here. Our post on evaluating RAG with Ragas shows how to automate that. If the reranking quality is not enough, a late-interaction reranker such as ColBERT is an alternative to the semantic ranker.

If your knowledge lives in SharePoint and Microsoft 365, also look at SharePoint lists as knowledge sources in Copilot Studio. For agents that use the index as a tool, see our guide to the Microsoft Agent Framework.

We build RAG systems on Azure for companies in Austria, Germany and Switzerland, from document preparation to evaluation. If you are planning one, our AI consulting is a good place to start, or get in touch directly.

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms
Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.