2025-11-18: Agentic knowledge base with Bing + SharePoint options

This release upgrades the agentic retrieval feature to use Azure AI Search's new Knowledge Base functionality, which provides enhanced intelligent document retrieval with configurable reasoning levels and support for multiple knowledge sources including web and SharePoint.

Knowledge Base changes

• Upgraded from Knowledge Agent to Knowledge Base: The feature now uses the new KnowledgeBaseRetrievalClient from Azure AI Search, replacing the legacy KnowledgeAgentRetrievalClient
• Added multiple reasoning level options: Choose between minimal, low, or medium reasoning effort for retrieval
  • Minimal: Most similar to current search() call, where we bring our own query planning to generate a single query
  • Low: Adds a query planning step that's built into AI Search and can suggest multiple queries for complex queries
  • Medium: Adds an optional iteration step if an AI Search model determines results aren't good enough

The feature supports three kinds of sources:

• Search Index (default): Searches documents in your Azure AI Search index. We currently only support one index, but you can modify the code to bring in additional indices if desired. We may add a YAML config for multiple indices in future.
• Web Source (optional): Enables searching the public web for information
  • ⚠️ Requires answer synthesis mode (disables streaming, follow-up questions, and some LLM customizations)
  • ⚠️ Microsoft Data Protection Addendum doesn't apply to data sent to Web Knowledge Source
• SharePoint Source (optional): Searches SharePoint documents using on-behalf-of flow
  • Requires authentication to be enabled
  • Requires users to have a Microsoft Copilot license

Environment variable changes

New environment variables for deployment configuration:

• USE_AGENTIC_KNOWLEDGEBASE: Enable the knowledge base feature (replaces USE_AGENTIC_RETRIEVAL)
• AZURE_SEARCH_KNOWLEDGEBASE_RETRIEVAL_REASONING_EFFORT: Set default reasoning level (minimal, low, or medium)
• USE_WEB_SOURCE: Enable web knowledge source
• USE_SHAREPOINT_SOURCE: Enable SharePoint knowledge source
• AZURE_OPENAI_KNOWLEDGEBASE_DEPLOYMENT: Specify the deployment for knowledge base
• AZURE_OPENAI_KNOWLEDGEBASE_MODEL: Specify the model (default: gpt-4.1-mini)
• AZURE_OPENAI_KNOWLEDGEBASE_MODEL_VERSION: Specify the model version

⚠️ If you were using the previous agentic retrieval feature (USE_AGENTIC_RETRIEVAL), you must change USE_AGENTIC_RETRIEVAL → USE_AGENTIC_KNOWLEDGEBASE

UI enhancements

• Developer settings: New options for toggling sources and changing reasoning level.

• Thought Process: Now includes:
  • Token usage for the planning process
  • Generated search queries
  • Activity details for each source citation

Documentation

See docs/agentic_retrieval.md for complete deployment and configuration instructions.

---

All changes included in this release

• Comment out the functions by default by @pamelafox in #2827
• Comment out final function in azure.yaml by @pamelafox in #2829
• Fixes issue with user uploaded files by @pamelafox in #2830
• Add Knowledge Base support by @mattgotteiner in #2835

Full Changelog: 2025-11-12...2025-11-18

2025-11-12: Cloud ingestion pipeline with custom skills (Azure Functions)

We're excited to announce a new cloud ingestion option that enables scalable, automated document processing using Azure Functions as custom skills within an Azure AI Search indexer pipeline. This repo previously supported an "integrated vectorization" strategy that used built-in skills for document extraction and splitting, but this new approach uses custom skills that use our own approach to document extraction and splitting. You can now use either local ingestion or cloud ingestion and get the same results.

This diagram shows the overall ingestion pipeline:

The indexer uses this skill sequence:

1. Document Extractor - Parses source documents (PDFs, HTML, Office files, images) and extracts text, tables, and figure metadata
2. Figure Processor - Enriches figures with AI-generated descriptions, uploads images to Blob Storage, and computes embeddings (when multimodal is enabled)
3. Shaper Skill - Consolidates enriched data from multiple processing paths into a unified document structure
4. Text Processor - Merges figures with text, performs sentence-aware chunking, and generates embeddings for vector search

The implementation required significant refactoring to maximize code reuse between local and cloud ingestion:

• Extracted shared setup logic into servicesetup.py
• Created standalone modules for figure processing (figureprocessor.py) and text processing (textprocessor.py)
• Bicep templates for Function App deployment with proper authentication

To enable the new feature, see the this guide. Please provide feedback in the issue tracker.

What's Changed

• Cloud ingestion strategy with prepdocs as custom skillset for Azure AI Search Blob Indexer by @pamelafox in #2819

Full Changelog: 2025-11-03...2025-11-12

2025-11-03: Port from AzureOpenAI to OpenAI constructors

Thanks to a recent improvement in the openai Python SDK and changes in the Azure OpenAI endpoint, we can now use OpenAI to connect to AzureOpenAI endpoint. The new Azure OpenAI endpoint is at "/openai/v1/" and does not accept an API version parameter, so we were able to remove the API version parameter entirely from this codebase, and simplify the code to only use the OpenAI() constructor.

If you merge in this change and see an OpenAI connection error locally, please make sure you have upgraded your local openai package version. You must have the version that accepts a token provider callback as the api_key parameter, otherwise you'll get an error about incorrect key/resource.

What's Changed

• Migrate AzureOpenAI constructors to standard OpenAI client by @Copilot in #2752

Full Changelog: 2025-10-29...2025-11-03

2025-10-29: Support Python 3.14 and drop support for 3.9

Python 3.14 is now the default version of Python downloaded from Python.org. We've upgraded all packages so that they should work in Python 3.14 environments. We've dropped 3.9 since it is reaching end-of-life and since many packages dropped 3.9 support when they added 3.14 support (like Pillow). Please update your local development environments to run at least 3.10.

What's Changed

• Bump flask from 3.0.3 to 3.1.2 by @dependabot[bot] in #2803
• Bump httpx from 0.27.0 to 0.28.1 by @dependabot[bot] in #2810
• Bump @types/dom-speech-recognition from 0.0.4 to 0.0.7 in /app/frontend by @dependabot[bot] in #2807
• Add Python 3.14 support and drop Python 3.9 by @Copilot in #2787
• Bump click from 8.1.8 to 8.3.0 in the pallets group by @dependabot[bot] in #2808
• Bump the github-actions group across 1 directory with 4 updates by @dependabot[bot] in #2783

Full Changelog: 2025-10-28...2025-10-29

2025-10-28: Upgraded ACLs to use Azure AI Search Query-time ACL

This release makes a significant change to our optional ACL feature, which builds on the login feature.

Previously we implemented access controls by storing oids and groups in the search index for each chunk, and dynamically building a search filter based off the oids/groups in the logged in user's token.
Now, Azure AI Search has built-in support to filter documents based off a user's token, so we keep the same index setup, but we delegate the filtering to the AI Search query itself.
If you already have the ACL feature enabled, you should be able to pull in the new changes, run "azd up", and have prepdocs setup the new permission filtering option on the index. If you have the AZURE_ENABLE_GLOBAL_DOCUMENT_ACCESS option enabled, then you'll also need to run the migration script to explicitly add a global identifier to the ACLs for those documents. Please also ensure that your azd environment has AZURE_ENFORCE_ACCESS_CONTROL, as we discovered some code paths weren't requiring that before (but do now).

If you are new to using the feature, please read through the login and ACLs guide to learn more.

What's Changed

• Better support for coding agents by @pamelafox in #2725
• added polish locales by @michuhu in #2729
• Bump jiter from 0.8.2 to 0.11.0 by @dependabot[bot] in #2731
• Bump msal from 1.30.0 to 1.33.0 by @dependabot[bot] in #2713
• Resolve issues with colliding private endpoint names, and redundant private endpoints by @kachihro in #2740
• Add missing set in command by @pamelafox in #2746
• Add tests for auth_init.py by @pamelafox in #2741
• Bump oauthlib from 3.2.2 to 3.3.1 by @dependabot[bot] in #2747
• Sanitize image markdown in sources by @pamelafox in #2765
• Restrict locations due to App Insights components availability by @pamelafox in #2764
• Add 'australiaeast' to allowed regions by @kachihro in #2770
• Bump multidict from 6.0.5 to 6.7.0 by @dependabot[bot] in #2768
• Group telemetry together for pip updates by @pamelafox in #2792
• Bump prismjs and react-syntax-highlighter in /app/frontend by @dependabot[bot] in #2791
• Bump the telemetry group with 21 updates by @dependabot[bot] in #2793
• Add dependabot group for pydantic by @pamelafox in #2795
• Bump markupsafe from 2.1.5 to 3.0.3 by @dependabot[bot] in #2794
• Bump vite from 5.4.19 to 6 in /app/frontend by @dependabot[bot] in #2782
• Bump pypdf from 6.0.0 to 6.1.3 in /app/backend by @pamelafox in #2799
• Bump rapidfuzz from 3.13.0 to 3.14.1 by @dependabot[bot] in #2798
• Bump asgiref from 3.8.1 to 3.10.0 by @dependabot[bot] in #2797
• Update link thats now 404ing by @pamelafox in #2801
• Bump the pydantic group with 2 updates by @dependabot[bot] in #2796
• Switch from security filter to built-in ACL enforcement by @mattgotteiner in #2771

New Contributors

• @michuhu made their first contribution in #2729
• @kachihro made their first contribution in #2740

Full Changelog: 2025-09-11...2025-10-28

2025-09-11: Upgrade to latest AI Search Agentic Retrieval API

The Azure AI Search team recently announced improvements to the agentic retrieval API. We've upgraded the repo to use the latest API, which brings the benefit that the retrieved data now includes all the searchable fields, improving the compatibility of the agentic retrieval feature with other repo features (multimodal, ACLs). Our repo does not currently use the multi-index query planning or answer synthesis, but you may consider those in your forks.

This release also includes a bug fix for user login on Container Apps and fixes for the recent multimodal feature.

What's Changed

• feat: Add extra search index fields to Knowledge Agent response by @taylorn-ai in #2696
• Bump tenacity from 9.0.0 to 9.1.2 by @dependabot[bot] in #2700
• Bump regex from 2024.11.6 to 2025.7.34 by @dependabot[bot] in #2701
• Bump click from 8.1.7 to 8.1.8 by @dependabot[bot] in #2698
• Bump azure-core from 1.30.2 to 1.35.0 by @dependabot[bot] in #2697
• Bump h2 from 4.1.0 to 4.3.0 in /app/backend by @dependabot[bot] in #2705
• Add comparison with microsoft/azurechat to other samples documentation by @Copilot in #2703
• Update chat/ask prompts for improved consistency, run multimodal evals by @pamelafox in #2709
• Adjust defaults for multimodal-related parameters to work for default deployments by @pamelafox in #2717
• Fix ingestion for case when no images field exists by @pamelafox in #2719
• Remove pipeline section from azure.yaml by @pamelafox in #2720
• Pin devcontainer to bookworm by @pamelafox in #2722
• Upgrade minimum node version on pipelines by @pamelafox in #2721
• Bump the github-actions group with 3 updates by @dependabot[bot] in #2716
• Bump rich from 13.9.4 to 14.1.0 by @dependabot[bot] in #2714
• Upgrade to latest version of azure-search-documents and agentic retrieval API by @pamelafox in #2723
• Add missing RBAC role for token storage container when using container apps by @pamelafox in #2724

New Contributors

• @taylorn-ai made their first contribution in #2696

Full Changelog: 2025-08-29...2025-09-11

2025-08-29: New approach to multimodal RAG

This release introduces a large change, a new approach to multimodal RAG based off learnings from our original vision feature:

The new multimodal approach affects both data ingestion and the RAG flow.

During data ingestion, the prepdocs script will:

• extract images (using Azure Document Intelligence) and stores them separately in Azure Blob storage
• compute embeddings of extracted images using Azure AI Vision
• Use LLM to describe the images inside the text chunk, and embed that description inside the text chunk
• associate each text chunk in the Azure AI Search index with any nearby images

During the RAG flow, the Chat and Ask approaches will:

• [Optionally] perform a multivector search on both text and image embeddings
• [Optionally] send images associated with search results to the multimodal LLM

Here's what it looks like in the UI:

For more information:

• Read the multimodal guide in the docs
• See the Pull Request
• Watch a demo video

This is a significant change, so it will be difficult to merge into existing forks of the repository. We know that developers like when it is easy to merge in new features, but we realized we needed to do some refactoring to achieve the goals. If you're merging into an existing branch, the best approach is probably to review the PR to understand the scope of the changes. Notably, we removed the separate *Vision approaches, and integrated multimodal directly into the main Chat/Ask approaches.

For those of you who are using integrated vectorization or the new agentic retrieval feature, the multimodal support is not yet fully compatible with those, but we will prioritize compatibility for a release in the near future.

What's Changed

• Add GPT-5 evals and "minimal" to reasoning dropdown by @pamelafox in #2671
• Adding custom debug chat mode for GitHub Copilot Agent mode development by @pamelafox in #2672
• Use lowest reasoning effort appropriate for a model by @pamelafox in #2673
• Improved custom chat mode and Copilot instructions file by @pamelafox in #2681
• Update error message to be platform agnostic by @pamelafox in #2682
• Bump Azure/setup-azd from 2.1.0 to 2.2.0 in the github-actions group by @dependabot[bot] in #2669
• Add additional eval results for gpt-5-mini by @pamelafox in #2683
• Bump actions/checkout from 4 to 5 in the github-actions group by @dependabot[bot] in #2684
• Bump pypdf from 4.3.1 to 6.0.0 in /app/backend by @dependabot[bot] in #2674
• Add markdownlint extension by @pamelafox in #2689
• Hyperlink leads to random Korean betting website by @Daimler-Garay in #2691
• Fix a11y landmark issue and add Axe Playwright test by @pamelafox in #2687
• Add test coverage to CI workflow by @pamelafox in #2690
• Bump azure-monitor-opentelemetry from 1.6.1 to 1.6.13 by @dependabot[bot] in #2663
• Fix useAgenticRetrieval missing Japanese translations by @Copilot in #2694
• Initialize MSAL before use to fix auth regression by @pamelafox in #2685
• New approach to multimodal document ingestion by @pamelafox in #2558

New Contributors

• @Daimler-Garay made their first contribution in #2691

Full Changelog: 2025-08-07...2025-08-29

2025-08-07: Support for GPT-5 model family

The repo now has support for the GPT-5 model family, just announced by OpenAI. The docs are updated to show how to use the reasoning models (gpt-5, gpt-5-mini, gpt-5-nano) and chat model (gpt-5-chat):

https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/reasoning.md

https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/deploy_features.md#using-different-chat-completion-models

Note that the regions are limited for the new models, and you must fill out a form for the full gpt-5, per the Azure OpenAI docs.

What's Changed

• Add support for GPT-5 model family by @pamelafox in #2667

Full Changelog: 2024-08-04b...2025-08-07

2025-08-04a: Support for o3 and o4-mini

The only significant feature change in this release is the support of the o3 and o4-mini models when using [reasoning.
The other changes were dependency upgrades and a new architecture documentation page.

What's Changed

• Bump urllib3 from 2.2.2 to 2.5.0 in /app/backend by @dependabot[bot] in #2581
• Update README.md with Foundry buttons, RAG Deep Dive link by @pamelafox in #2587
• Bump attrs from 24.2.0 to 25.3.0 by @dependabot[bot] in #2563
• Bump python-dotenv from 1.0.1 to 1.1.1 by @dependabot[bot] in #2588
• Bump requests from 2.32.3 to 2.32.4 in /app/backend by @dependabot[bot] in #2566
• Bump std-uritemplate from 2.0.3 to 2.0.5 by @dependabot[bot] in #2536
• Upgrade h11 dependency by @pamelafox in #2596
• Updates to add latest omni models, upgrade package lock by @pamelafox in #2597
• Bump soupsieve from 2.6 to 2.7 by @dependabot[bot] in #2601
• Bump aiohttp from 3.10.11 to 3.12.14 in /app/backend by @dependabot[bot] in #2606
• Remove conditional Azure login steps and simplify authentication in deployment workflows by @Copilot in #2625
• Fix Dependabot MSAL package upgrade by updating compatible versions by @Copilot in #2632
• Upgrade rapidfuzz from 3.12.1 to 3.13.0 to fix failed Dependabot PR #2504 by @Copilot in #2646
• Fix Vite 7.0.6 upgrade by updating @vitejs/plugin-react to v4.7.0 by @Copilot in #2630
• Add comprehensive Mermaid architecture diagrams for application documentation by @Copilot in #2653
• Revert vite to earlier version for node v20.14 compatibility by @pamelafox in #2657

New Contributors

• @Copilot made their first contribution in #2625

Full Changelog: 2025-06-03...2025-08-04

2025-08-04b: Private networking for Container Apps + P2S VPN Gateway

This release updates the private networking feature to add support for Azure Container Apps, the default deployment host.
It also adds an optional P2S VPN Gateway (secured with Entra ID) with an Azure Private DNS resolver, so that developers can test and deploy from their own machines.

Please open an issue if you try out the feature and encounter any issues. Please make sure you are using additional security auditing mechanism to confirm the deployment meets the need of your organization, such as Microsoft Defender for Cloud.

What's Changed

• Private endpoint support for container apps by @pamelafox in #2322
• Add Bicep description for infra/private-endpoints.bicep by @Copilot in #2665
• Update deploy_private.md docs to reflect new feature by @pamelafox in #2666

Full Changelog: 2025-08-04...2024-08-04b