Merge branch 'main' into visionv2

Author: pamelafox

README.md

@@ -51,7 +51,7 @@ This template, the application code and configuration it contains, has been buil
 
 [📺 Watch a video overview of the app.](https://youtu.be/3acB0OWmLvM)
 
-This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4o-mini), and Azure AI Search for data indexing and retrieval.
+This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4.1-mini), and Azure AI Search for data indexing and retrieval.
 
 The repo includes sample data so it's ready to try end to end. In this sample application we use a fictitious company called Contoso Electronics, and the experience allows its employees to ask questions about the benefits, internal policies, as well as job descriptions and roles.
 
@@ -258,9 +258,11 @@ You can find extensive documentation in the [docs](docs/README.md) folder:
     - [Multimodal](docs/multimodal.md)
     - [Reasoning](docs/reasoning.md)
     - [Private endpoints](docs/deploy_private.md)
+    - [Agentic retrieval](docs/agentic_retrieval.md)
   - [Sharing deployment environments](docs/sharing_environments.md)
 - [Local development](docs/localdev.md)
 - [Customizing the app](docs/customization.md)
+- [HTTP Protocol](docs/http_protocol.md)
 - [Data ingestion](docs/data_ingestion.md)
 - [Evaluation](docs/evaluation.md)
 - [Safety evaluation](docs/safety_evaluation.md)

app/backend/requirements.txt

@@ -342,7 +342,7 @@ pyjwt==2.10.1
     # via
     #   -r requirements.in
     #   msal
-pymupdf==1.25.1
+pymupdf==1.26.0
     # via -r requirements.in
 pypdf==4.3.1
     # via -r requirements.in
@@ -398,7 +398,7 @@ types-html5lib==1.1.11.20241018
     # via types-beautifulsoup4
 types-pillow==10.2.0.20240822
     # via -r requirements.in
-typing-extensions==4.12.2
+typing-extensions==4.13.2
     # via
     #   -r requirements.in
     #   azure-ai-documentintelligence

docs/README.md

@@ -14,12 +14,14 @@ These are advanced topics that are not necessary for a basic deployment.
     - [Login and access control](login_and_acl.md)
     - [GPT-4 Turbo with Vision](gpt4v.md)
     - [Private endpoints](deploy_private.md)
+    - [Agentic retrieval](agentic_retrieval.md)
   - [Sharing deployment environments](sharing_environments.md)
 - [Local development](localdev.md)
 - [Customizing the app](customization.md)
+- [HTTP Protocol](http_protocol.md)
+- [Data ingestion](data_ingestion.md)
 - [Evaluation](docs/evaluation.md)
 - [Safety evaluation](safety_evaluation.md)
-- [Data ingestion](data_ingestion.md)
 - [Monitoring with Application Insights](monitoring.md)
 - [Productionizing](productionizing.md)
 - [Alternative RAG chat samples](other_samples.md)

docs/agentic_retrieval.md

@@ -10,7 +10,7 @@ See the agentic retrieval documentation.
 
 ### Prerequisites
 
-* A deployment of any of the supported agentic retrieval models in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o-mini deployment from your Azure OpenAI deployments page.
+* A deployment of any of the supported agentic retrieval models in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4.1-mini deployment from your Azure OpenAI deployments page.
 
 ### Deployment
 
@@ -24,14 +24,14 @@ See the agentic retrieval documentation.
 
 2. **(Optional) Set the agentic retrieval model**
 
-   You can configure which model agentic retrieval uses. By default, gpt-4o-mini is used
+   You can configure which model agentic retrieval uses. By default, gpt-4.1-mini is used.
 
-   For gpt-4o:
+   To change the model, set the following environment variables appropriately:
 
    ```shell
    azd env set AZURE_OPENAI_SEARCHAGENT_DEPLOYMENT searchagent
-   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL gpt-4o
-   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL_VERSION 2024-11-20
+   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL gpt-4.1-mini
+   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL_VERSION 2025-04-14
    ```
 
 3. **Update the infrastructure and application:**

docs/customization.md

@@ -28,7 +28,7 @@ The frontend is built using [React](https://reactjs.org/) and [Fluent UI compone
 
 ## Customizing the backend
 
-The backend is built using [Quart](https://quart.palletsprojects.com/), a Python framework for asynchronous web applications. The backend code is stored in the `app/backend` folder. The frontend and backend communicate using the [AI Chat HTTP Protocol](https://aka.ms/chatprotocol).
+The backend is built using [Quart](https://quart.palletsprojects.com/), a Python framework for asynchronous web applications. The backend code is stored in the `app/backend` folder. The frontend and backend communicate over HTTP using JSON or streamed NDJSON responses. Learn more in the [HTTP Protocol guide](http_protocol.md).
 
 ### Chat/Ask tabs
 
@@ -46,7 +46,7 @@ The prompts are currently tailored to the sample data since they start with "Ass
 
 ##### Chat with vision
 
-If you followed the instructions in [docs/gpt4v.md](gpt4v.md) to enable a GPT Vision model and then select "Use GPT vision model", then the chat tab will use the `chatreadretrievereadvision.py` approach instead. This approach is similar to the `chatreadretrieveread.py` approach, with a few differences:
+If you followed the instructions in [the GPT vision guide](gpt4v.md) to enable the vision approach and the "Use GPT vision model" option is selected, then the chat tab will use the `chatreadretrievereadvision.py` approach instead. This approach is similar to the `chatreadretrieveread.py` approach, with a few differences:
 
 1. Step 1 is the same as before, except it uses the GPT-4 Vision model instead of the default GPT-3.5 model.
 2. For this step, it also calculates a vector embedding for the user question using [the Computer Vision vectorize text API](https://learn.microsoft.com/azure/ai-services/computer-vision/how-to/image-retrieval#call-the-vectorize-text-api), and passes that to the Azure AI Search to compare against the `imageEmbeddings` fields in the indexed documents. For each matching document, it downloads the image blob and converts it to a base 64 encoding.
@@ -65,7 +65,7 @@ The prompt for step 2 is currently tailored to the sample data since it starts w
 
 #### Ask with vision
 
-If you followed the instructions in [docs/gpt4v.md](gpt4v.md) to enable the GPT-4 Vision model and then select "Use GPT vision model", then the ask tab will use the `retrievethenreadvision.py` approach instead. This approach is similar to the `retrievethenread.py` approach, with a few differences:
+If you followed the instructions in [the GPT vision guide](gpt4v.md) to enable the vision approach and the "Use GPT vision model" option is selected, then the ask tab will use the `retrievethenreadvision.py` approach instead. This approach is similar to the `retrievethenread.py` approach, with a few differences:
 
 1. For this step, it also calculates a vector embedding for the user question using [the Computer Vision vectorize text API](https://learn.microsoft.com/azure/ai-services/computer-vision/how-to/image-retrieval#call-the-vectorize-text-api), and passes that to the Azure AI Search to compare against the `imageEmbeddings` fields in the indexed documents. For each matching document, it downloads the image blob and converts it to a base 64 encoding.
 2. When it combines the search results and user question, it includes the base 64 encoded images, and sends along both the text and images to the GPT4 Vision model (similar to this [documentation example](https://platform.openai.com/docs/guides/vision/quick-start)). The model generates a response that includes citations to the images, and the UI renders the base64 encoded images when a citation is clicked.

docs/deploy_existing.md

@@ -26,8 +26,8 @@ You should set these values before running `azd up`. Once you've set them, retur
 1. Run `azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}`
 1. Run `azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}`
 1. Run `azd env set AZURE_OPENAI_LOCATION {Location of existing OpenAI service}`
-1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4o-mini'.
-1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4o-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4.1-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4.1-mini'.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION {Version string for existing chat deployment}`. Only needed if your chat deployment model version is not the default '2024-07-18'. You definitely need to change this if you changed the model.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU {Name of SKU for existing chat deployment}`. Only needed if your chat deployment SKU is not the default 'Standard', like if it is 'GlobalStandard' instead.
 1. Run `azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing embedding deployment}`. Only needed if your embeddings deployment is not the default 'embedding'.

docs/deploy_features.md

@@ -24,7 +24,7 @@ You should typically enable these features before running `azd up`. Once you've
 
 ## Using different chat completion models
 
-As of late March 2025, the default chat completion model is `gpt-4o-mini`. If you deployed this sample before that date, the default model is `gpt-3.5-turbo`. You can change the chat completion model to any Azure OpenAI chat model that's available in your Azure OpenAI resource region by following these steps:
+As of early June 2025, the default chat completion model is `gpt-4.1-mini`. If you deployed this sample before that date, the default model is `gpt-3.5-turbo` or `gpt-4o-mini`. You can change the chat completion model to any Azure OpenAI chat model that's available in your Azure OpenAI resource region by following these steps:
 
 1. To set the name of the deployment, run this command with a unique name in your Azure OpenAI account. You can use any deployment name, as long as it's unique in your Azure OpenAI account. For convenience, many developers use the same deployment name as the model name, but this is not required.
 
@@ -40,24 +40,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the GPT model to a different [available model](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate model name.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
     ```bash
-    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4.1-mini
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o-mini
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash
@@ -66,24 +72,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the Azure OpenAI model version from the [available versions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate version string.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
     ```bash
-    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2025-04-14
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-05-13
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-07-18
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash

docs/gpt4v.md

@@ -23,7 +23,7 @@ For more details on how this feature works, read [this blog post](https://techco
 * The ability to deploy a gpt-4o model in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o deployment from your Azure OpenAI deployments page.
 * Ensure that you can deploy the Azure OpenAI resource group in [a region and deployment SKU where all required components are available](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability):
   * Azure OpenAI models
-    * gpt-4o-mini
+    * gpt-4.1-mini
     * text-embedding-3-large
     * gpt-4o (for vision/evaluation features)
   * [Azure AI Vision](https://learn.microsoft.com/azure/ai-services/computer-vision/)