Update the docs and dev container based off recent feedback (#2224)

* Updating docs

* Update requirements.txt

* Update diagram

* Add typing extensions explicitly

Author: pamelafox

.devcontainer/devcontainer.json

@@ -7,7 +7,8 @@
             "version": "18",
             "nodeGypDependencies": false
         },
-        "ghcr.io/devcontainers/features/azure-cli:1.0.8": {},
+        "ghcr.io/devcontainers/features/azure-cli:1.2.5": {},
+        "ghcr.io/devcontainers/features/docker-in-docker:2": {},
         "ghcr.io/azure/azure-dev/azd:latest": {}
     },
     "customizations": {

README.md

@@ -74,7 +74,7 @@ The repo includes sample data so it's ready to try end to end. In this sample ap
 
 **IMPORTANT:** In order to deploy and run this example, you'll need:
 
-- **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free/cognitive-search/) and you'll get some free Azure credits to get started. See [guide to deploying with the free trial](docs/deploy_lowcost.md).
+- **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free/cognitive-search/) and you'll get some free Azure credits to get started. See [guide to deploying with the free trial](docs/deploy_freetrial.md).
 - **Azure account permissions**:
   - Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](docs/deploy_existing.md#resource-group).
   - Your Azure account also needs `Microsoft.Resources/deployments/write` permissions on the subscription level.
@@ -172,7 +172,7 @@ The steps below will provision Azure resources and deploy the application code t
 
     Enter a name that will be used for the resource group.
     This will create a new folder in the `.azure` folder, and set it as the active environment for any calls to `azd` going forward.
-1. (Optional) This is the point where you can customize the deployment by setting environment variables, in order to [use existing resources](docs/deploy_existing.md), [enable optional features (such as auth or vision)](docs/deploy_features.md), or [deploy to free tiers](docs/deploy_lowcost.md).
+1. (Optional) This is the point where you can customize the deployment by setting environment variables, in order to [use existing resources](docs/deploy_existing.md), [enable optional features (such as auth or vision)](docs/deploy_features.md), or [deploy low-cost options](docs/deploy_lowcost.md), or [deploy with the Azure free trial](docs/deploy_freetrial.md).
 1. Run `azd up` - This will provision Azure resources and deploy this sample to those resources, including building the search index based on the files found in the `./data` folder.
     - **Important**: Beware that the resources created by this command will incur immediate costs, primarily from the AI Search resource. These resources may accrue costs even if you interrupt the command before it is fully executed. You can run `azd down` or delete the resources manually to avoid unnecessary spending.
     - You will be prompted to select two locations, one for the majority of resources and one for the OpenAI resource, which is currently a short list. That location list is based on the [OpenAI model availability table](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability) and may become outdated as availability changes.

app/backend/requirements.in

@@ -16,7 +16,6 @@ aiohttp
 azure-monitor-opentelemetry
 opentelemetry-instrumentation-asgi
 opentelemetry-instrumentation-httpx
-opentelemetry-instrumentation-requests
 opentelemetry-instrumentation-aiohttp-client
 opentelemetry-instrumentation-openai
 msal
@@ -32,3 +31,4 @@ msgraph-sdk==1.1.0
 openai-messages-token-helper
 python-dotenv
 rich
+typing-extensions

app/backend/requirements.txt

@@ -270,9 +270,7 @@ opentelemetry-instrumentation-openai==0.27.0
 opentelemetry-instrumentation-psycopg2==0.47b0
     # via azure-monitor-opentelemetry
 opentelemetry-instrumentation-requests==0.47b0
-    # via
-    #   -r requirements.in
-    #   azure-monitor-opentelemetry
+    # via azure-monitor-opentelemetry
 opentelemetry-instrumentation-urllib==0.47b0
     # via azure-monitor-opentelemetry
 opentelemetry-instrumentation-urllib3==0.47b0
@@ -413,6 +411,7 @@ types-pillow==10.2.0.20240822
     # via -r requirements.in
 typing-extensions==4.12.2
     # via
+    #   -r requirements.in
     #   azure-ai-documentintelligence
     #   azure-core
     #   azure-cosmos

docs/deploy_freetrial.md

@@ -0,0 +1,43 @@
+# RAG chat: Deploying with a free trial account
+
+If you have just created an Azure free trial account and are using the free trial credits,
+there are several modifications you need to make, due to restrictions on the free trial account.
+
+Follow these instructions *before* you run `azd up`.
+
+## Accomodate for low OpenAI quotas
+
+The free trial accounts currently get a max of 1K TPM (tokens-per-minute), whereas our Bicep templates try to allocate 30K TPM.
+
+To reduce the TPM allocation, run these commands:
+
+```shell
+azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 1
+azd env set AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY 1
+```
+
+Alternatively, if you have an OpenAI.com account, you can use that instead:
+
+```shell
+azd env set OPENAI_HOST openai
+azd env set OPENAI_ORGANIZATION {Your OpenAI organization}
+azd env set OPENAI_API_KEY {Your OpenAI API key}
+```
+
+## Accomodate for Azure Container Apps restrictions
+
+By default, this project deploys to Azure Container Apps, using a remote build process that builds the Docker image in the cloud.
+Unfortunately, free trial accounts cannot use that remote build process.
+
+You have two options:
+
+1. Comment out or delete `remoteBuild: true` in `azure.yaml`, and make sure you have Docker installed in your environment.
+
+2. Deploy using App Service instead:
+
+    * Comment out `host: containerapp` and uncomment `host: appservice` in the [azure.yaml](../azure.yaml) file.
+    * Set the deployment target to `appservice`:
+
+        ```shell
+        azd env set DEPLOYMENT_TARGET appservice
+        ```

docs/deploy_lowcost.md

@@ -21,11 +21,24 @@ However, if your goal is to minimize costs while prototyping your application, f
     Enter a name that will be used for the resource group.
     This will create a new folder in the `.azure` folder, and set it as the active environment for any calls to `azd` going forward.
 
-1. Use the free tier of App Service:
+1. Switch from Azure Container Apps to the free tier of Azure App Service:
 
-    ```shell
-    azd env set AZURE_APP_SERVICE_SKU F1
-    ```
+    Azure Container Apps has a consumption-based pricing model that is very low cost, but it is not free, plus Azure Container Registry costs a small amount each month.
+
+    To deploy to App Service instead:
+
+    * Comment out `host: containerapp` and uncomment `host: appservice` in the [azure.yaml](../azure.yaml) file.
+    * Set the deployment target to `appservice`:
+
+        ```shell
+        azd env set DEPLOYMENT_TARGET appservice
+        ```
+
+    * Set the App Service SKU to the free tier:
+
+        ```shell
+        azd env set AZURE_APP_SERVICE_SKU F1
+        ```
 
     Limitation: You are only allowed a certain number of free App Service instances per region. If you have exceeded your limit in a region, you will get an error during the provisioning stage. If that happens, you can run `azd down`, then `azd env new` to create a new environment with a new region.
 
@@ -88,7 +101,7 @@ However, if your goal is to minimize costs while prototyping your application, f
     Application Insights is quite inexpensive already, so turning this off may not be worth the costs saved,
     but it is an option for those who want to minimize costs.
 
-1. Use OpenAI.com instead of Azure OpenAI: This is only a necessary step for Azure free/student accounts, as they do not currently have access to Azure OpenAI.
+1. Use OpenAI.com instead of Azure OpenAI: This should not be necessary, as the costs are same for both services, but you may need this step if your account does not have access to Azure OpenAI for some reason.
 
     ```shell
     azd env set OPENAI_HOST openai

docs/gpt4v.md

@@ -1,33 +1,34 @@
 # RAG chat: Using GPT vision model with RAG approach
 
-This repository now includes an example of integrating a GPT Vision model with Azure AI Search. This feature enables indexing and searching images and graphs, such as financial documents, in addition to text-based content, and then sending the retrieved content to the GPT model for response generation.
+This repository includes an optional feature that uses the GPT vision model to generate responses based on retrieved content. This feature is useful for answering questions based on the visual content of documents, such as photos and charts.
 
-## Feature Overview
+## How it works
 
-- **Document Handling:** Source documents are split into pages and saved as PNG files in blob storage. Each file's name and page number are embedded for reference.
-- **Data Extraction:** Text data is extracted using OCR.
-- **Data Indexing:** Text and image embeddings, generated using Azure AI Vision ([Azure AI Vision Embeddings](https://learn.microsoft.com/azure/ai-services/computer-vision/how-to/image-retrieval)), are indexed in Azure AI Search along with the raw text.
-- **Search and Response:** Searches can be conducted using vectors or hybrid methods. Responses are generated by GPT vision model based on the retrieved content.
+When this feature is enabled, the following changes are made to the application:
 
-## Getting Started
+* **Search index**: We added a new field to the Azure AI Search index to store the embedding returned by the multimodal Azure AI Vision API (while keeping the existing field that stores the OpenAI text embeddings).
+* **Data ingestion**: In addition to our usual PDF ingestion flow, we also convert each PDF document page to an image, store that image with the filename rendered on top, and add the embedding to the index.
+* **Question answering**: We search the index using both the text and multimodal embeddings. We send both the text and the image to gpt-4o, and ask it to answer the question based on both kinds of sources.
+* **Citations**: The frontend displays both image sources and text sources, to help users understand how the answer was generated.
 
-### Prerequisites
+For more details on how this feature works, read [this blog post](https://techcommunity.microsoft.com/blog/azuredevcommunityblog/integrating-vision-into-rag-applications/4239460) or watch [this video](https://www.youtube.com/live/C3Zq3z4UQm4?si=SSPowBBJoTBKZ9WW&t=89).
+
+## Using the feature
 
-- Create a [Computer Vision account in Azure Portal first](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision), so that you can agree to the Responsible AI terms for that resource. You can delete that account after agreeing.
-- The ability to deploy a gpt-4o model in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o deployment from your Azure OpenAI deployments page.
-- Ensure that you can deploy the Azure OpenAI resource group in [a region where all required components are available](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability):
-  - Azure OpenAI models
-    - gpt-35-turbo
-    - text-embedding-ada-002
-    - gpt-4o
-  - [Azure AI Vision](https://learn.microsoft.com/azure/ai-services/computer-vision/)
+### Prerequisites
 
-### Setup and Usage
+* Create a [AI Vision account in Azure Portal first](https://ms.portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision), so that you can agree to the Responsible AI terms for that resource. You can delete that account after agreeing.
+* The ability to deploy a gpt-4o model in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o deployment from your Azure OpenAI deployments page.
+* Ensure that you can deploy the Azure OpenAI resource group in [a region where all required components are available](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability):
+  * Azure OpenAI models
+    * gpt-35-turbo
+    * text-embedding-ada-002
+    * gpt-4o
+  * [Azure AI Vision](https://learn.microsoft.com/azure/ai-services/computer-vision/)
 
-1. **Update repository:**
-   Pull the latest changes.
+### Deployment
 
-2. **Enable GPT vision approach:**
+1. **Enable GPT vision approach:**
 
    First, make sure you do *not* have integrated vectorization enabled, since that is currently incompatible:
 
@@ -41,19 +42,17 @@ This repository now includes an example of integrating a GPT Vision model with A
    azd env set USE_GPT4V true
    ```
 
-   When set, that flag will provision a Computer Vision resource and gpt-4o model, upload image versions of PDFs to Blob storage, upload embeddings of images in a new `imageEmbedding` field, and enable the vision approach in the UI.
+   When set, that flag will provision a Azure AI Vision resource and gpt-4o model, upload image versions of PDFs to Blob storage, upload embeddings of images in a new `imageEmbedding` field, and enable the vision approach in the UI.
 
-3. **Clean old deployments (optional):**
+2. **Clean old deployments (optional):**
    Run `azd down --purge` for a fresh setup.
 
-4. **Start the application:**
+3. **Start the application:**
    Execute `azd up` to build, provision, deploy, and initiate document preparation.
 
-5. **Web Application Usage:**
+4. **Try out the feature:**
     ![GPT4V configuration screenshot](./images/gpt4v.png)
-   - Access the developer options in the web app and select "Use GPT vision model".
-   - Sample questions will be updated for testing.
-   - Interact with the questions to view responses.
-   - The 'Thought Process' tab shows the retrieved data and its processing by the GPT vision model.
-
-Feel free to explore and contribute to enhancing this feature. For questions or feedback, use the repository's issue tracker.
+   * Access the developer options in the web app and select "Use GPT vision model".
+   * New sample questions will show up in the UI that are based on the sample financial document.
+   * Try out a question and see the answer generated by the GPT vision model.
+   * Check the 'Thought process' and 'Supporting content' tabs.

docs/images/diagram_prepdocs.png

@@ -123,13 +123,14 @@ The following instructions explain how to setup the two apps using the Azure Por
   - In the **Name** section, enter a meaningful application name. This name will be displayed to users of the app, for example `Azure Search OpenAI Chat Web App`.
   - Under **Supported account types**, select **Accounts in this organizational directory only**.
   - Under `Redirect URI (optional)` section, select `Single-page application (SPA)` in the combo-box and enter the following redirect URI:
-    - If you are running the sample locally, use `http://localhost:50505/redirect`.
-    - If you are running the sample on Azure, use the endpoint provided by `azd up`: `https://<your-endpoint>.azurewebsites.net/redirect`.
-    - If you are running the sample from Github Codespaces, use the Codespaces endpoint: `https://<your-codespace>-50505.app.github.dev/`
+    - If you are running the sample locally, add the endpoints `http://localhost:50505/redirect` and `http://localhost:5173/redirect`
+    - If you are running the sample on Azure, add the endpoints provided by `azd up`: `https://<your-endpoint>.azurewebsites.net/redirect`.
+    - If you are running the sample from Github Codespaces, add the Codespaces endpoint: `https://<your-codespace>-50505.app.github.dev/redirect`
 - Select **Register** to create the application
 - In the app's registration screen, find the **Application (client) ID**.
   - Run the following `azd` command to save this ID: `azd env set AZURE_CLIENT_APP_ID <Application (client) ID>`.
 - In the left hand menu, select **Authentication**.
+  - Under Web, add a redirect URI with the endpoint provided by `azd up`: `https://<your-endpoint>.azurewebsites.net/.auth/login/aad/callback`.
   - Under **Implicit grant and hybrid flows**, select **ID Tokens (used for implicit and hybrid flows)**
   - Select **Save**
 - In the left hand menu, select **API permissions**. You will add permission to access the **access_as_user** API on the server app. This permission is required for the [On Behalf Of Flow](https://learn.microsoft.com/entra/identity-platform/v2-oauth2-on-behalf-of-flow#protocol-diagram) to work.
@@ -138,6 +139,11 @@ The following instructions explain how to setup the two apps using the Azure Por
   - Ensure **Delegated permissions** is selected.
   - In the **Select permissions** section, select the **access_as_user** permission
   - Select **Add permissions**.
+- Stay in the **API permissions** section and select **Add a permission**.
+  - Select **Microsoft Graph**.
+  - Select **Delegated permissions**.
+  - Search for and select `User.Read`.
+  - Select **Add permissions**.
 
 #### Configure Server App Known Client Applications