Additional pointers to docs on private endpoints and load balancing (#1618)

* More instructions

* Updated TOC

* Markdown lint stuff

* More markdown fixes

Author: pamelafox

docs/README.md

@@ -9,6 +9,7 @@ These are advanced topics that are not necessary for a basic deployment.
     * [Enabling optional features](deploy_features.md)
         * [Login and access control](login_and_acl.md)
         * [GPT-4 Turbo with Vision](gpt4v.md)
+        * [Private endpoints](deploy_private.md)
 * [Debugging the app on App Service](appservice.md)
 * [Local development](localdev.md)
 * [App customization](customization.md)

docs/deploy_features.md

@@ -12,6 +12,8 @@ You should typically enable these features before running `azd up`. Once you've
 * [Enabling login and document level access control](#enabling-login-and-document-level-access-control)
 * [Enabling user document upload](#enabling-user-document-upload)
 * [Enabling CORS for an alternate frontend](#enabling-cors-for-an-alternate-frontend)
+* [Adding an OpenAI load balancer](#adding-an-openai-load-balancer)
+* [Deploying with private endpoints](#deploying-with-private-endpoints)
 * [Using local parsers](#using-local-parsers)
 
 ## Using GPT-4
@@ -52,11 +54,12 @@ Execute the following commands inside your terminal:
 
 > [!NOTE]
 > To revert back to GPT 3.5, run the following commands:
-> - `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT chat` to set the name of your old GPT 3.5 deployment.
-> - `azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-35-turbo` to set the name of your old GPT 3.5 model.
-> - `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 30` to set the capacity of your old GPT 3.5 deployment.
-> - `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 0613` to set the version number of your old GPT 3.5.
-> - `azd up` to update the provisioned resources.
+>
+> * `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT chat` to set the name of your old GPT 3.5 deployment.
+> * `azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-35-turbo` to set the name of your old GPT 3.5 model.
+> * `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_CAPACITY 30` to set the capacity of your old GPT 3.5 deployment.
+> * `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 0613` to set the version number of your old GPT 3.5.
+> * `azd up` to update the provisioned resources.
 >
 > Note that this does not delete your GPT-4 deployment; it just makes your application create a new or reuse an old GPT 3.5 deployment. If you want to delete it, you can go to your Azure OpenAI studio and do so.
 
@@ -86,7 +89,7 @@ By default, the deployed Azure web app uses the `text-embedding-ada-002` embeddi
     azd env set AZURE_OPENAI_EMB_DEPLOYMENT_VERSION 1
     ```
 
-3. When prompted during `azd up`, make sure to select a region for the OpenAI resource group location that supports the text-embedding-3 models. There are [limited regions available](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#embeddings-models).
+4. When prompted during `azd up`, make sure to select a region for the OpenAI resource group location that supports the text-embedding-3 models. There are [limited regions available](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#embeddings-models).
 
 If you have already deployed:
 
@@ -151,7 +154,6 @@ And then update existing search documents with the storage URL of the main Blob
 Going forward, all uploaded documents will have their `storageUrl` set in the search index.
 This is necessary to disambiguate user-uploaded documents from admin-uploaded documents.
 
-
 ## Enabling CORS for an alternate frontend
 
 By default, the deployed Azure web app will only allow requests from the same origin.  To enable CORS for a frontend hosted on a different origin, run:
@@ -166,6 +168,18 @@ For an alternate frontend that's written in Web Components and deployed to Stati
 on [using a different backend](https://github.com/Azure-Samples/azure-search-openai-javascript#using-a-different-backend).
 Both these repositories adhere to the same [HTTP protocol for RAG chat apps](https://github.com/Azure-Samples/ai-chat-app-protocol).
 
+## Adding an OpenAI load balancer
+
+As discussed in more details in our [productionizing guide](docs/productionizing.md), you may want to consider implementing a load balancer between OpenAI instances if you are consistently going over the TPM limit.
+Fortunately, this repository is designed for easy integration with other repositories that create load balancers for OpenAI instances. For seamless integration instructions with this sample, please check:
+
+* [Scale Azure OpenAI for Python with Azure API Management](https://learn.microsoft.com/azure/developer/python/get-started-app-chat-scaling-with-azure-api-management)
+* [Scale Azure OpenAI for Python chat using RAG with Azure Container Apps](https://learn.microsoft.com/azure/developer/python/get-started-app-chat-scaling-with-azure-container-apps)
+
+## Deploying with private endpoints
+
+It is possible to deploy this app with public access disabled, using Azure private endpoints and private DNS Zones. For more details, read [the private deployment guide](docs/deploy_private.md). That requires a multi-stage provisioning, so you will need to do more than just `azd up` after setting the environment variables.
+
 ## Using local parsers
 
 If you want to decrease the charges by using local parsers instead of Azure Document Intelligence, you can set environment variables before running the [data ingestion script](./data_ingestion.md). Note that local parsers will generally be not as sophisticated.

docs/deploy_private.md

@@ -8,41 +8,46 @@ If you want to disable public access when deploying the Chat App, you can do so
 Deploying with public access disabled adds additional cost to your deployment. Please see pricing for the following products:
 
 1. [Private Endpoints](https://azure.microsoft.com/pricing/details/private-link/)
-  1. The exact number of private endpoints created depends on the [optional features](./deploy_features.md) used.
+    1. The exact number of private endpoints created depends on the [optional features](./deploy_features.md) used.
 1. [Private DNS Zones](https://azure.microsoft.com/pricing/details/dns/)
 1. (Optional, but recommended)[Azure Virtual Machines](https://azure.microsoft.com/pricing/details/virtual-machines/windows/)
 1. (Optional, but recommended)[Azure Bastion](https://azure.microsoft.com/pricing/details/azure-bastion/)
 
 ## Environment variables controlling private access
 
 1. `AZURE_PUBLIC_NETWORK_ACCESS`: Controls the value of public network access on supported Azure resources. Valid values are 'Enabled' or 'Disabled'.
-  1. When public network access is 'Enabled', Azure resources are open to the internet.
-  1. When public network access is 'Disabled', Azure resources are only accessible over a virtual network.
+    1. When public network access is 'Enabled', Azure resources are open to the internet.
+    1. When public network access is 'Disabled', Azure resources are only accessible over a virtual network.
 1. `AZURE_USE_PRIVATE_ENDPOINT`: Controls deployment of [private endpoints](https://learn.microsoft.com/azure/private-link/private-endpoint-overview) which connect Azure resources to the virtual network.
-  1. When set to 'true', ensures private endpoints are deployed for connectivity even when `AZURE_PUBLIC_NETWORK_ACCESS` is 'Disabled'.
-  1. Note that private endpoints do not make the chat app accessible from the internet. Connections must be initiated from inside the virtual network.
+    1. When set to 'true', ensures private endpoints are deployed for connectivity even when `AZURE_PUBLIC_NETWORK_ACCESS` is 'Disabled'.
+    1. Note that private endpoints do not make the chat app accessible from the internet. Connections must be initiated from inside the virtual network.
 1. `AZURE_PROVISION_VM`: Controls deployment of a [virtual machine](https://learn.microsoft.com/azure/virtual-machines/overview) and [Azure Bastion](https://learn.microsoft.com/azure/bastion/bastion-overview). Azure Bastion allows you to securely connect to the virtual machine, without being connected virtual network. Since the virtual machine is connected to the virtual network, you are able to access the chat app.
-  1. You must set `AZURE_VM_USERNAME` and `AZURE_VM_PASSWORD` to provision the built-in administrator account with the virtual machine so you can log in through Azure Bastion.
-  1. By default, a server version of Windows is used for the VM. If you need to [enroll your device in Microsoft Intune](https://learn.microsoft.com/mem/intune/user-help/enroll-windows-10-device), you should use a desktop version of Windows by setting the following environment variables:
-    * `azd env set AZURE_VM_OS_PUBLISHER MicrosoftWindowsDesktop`
-    * `azd env set AZURE_VM_OS_OFFER Windows-11`
-    * `azd env set AZURE_VM_OS_VERSION win11-23h2-pro`
+    1. You must set `AZURE_VM_USERNAME` and `AZURE_VM_PASSWORD` to provision the built-in administrator account with the virtual machine so you can log in through Azure Bastion.
+    1. By default, a server version of Windows is used for the VM. If you need to [enroll your device in Microsoft Intune](https://learn.microsoft.com/mem/intune/user-help/enroll-windows-10-device), you should use a desktop version of Windows by setting the following environment variables:
+
+      * `azd env set AZURE_VM_OS_PUBLISHER MicrosoftWindowsDesktop`
+      * `azd env set AZURE_VM_OS_OFFER Windows-11`
+      * `azd env set AZURE_VM_OS_VERSION win11-23h2-pro`
 
 ## Recommended deployment strategy for private access
 
 1. Deploy the app with private endpoints enabled and public access enabled.
-```
-azd env set AZURE_USE_PRIVATE_ENDPOINT true
-azd env set AZURE_PUBLIC_NETWORK_ACCESS Enabled
-azd up
-```
-2. Validate that you can connect to the chat app and it's working as expected from the internet.
-3. Re-provision the app with public access disabled.
-```
-azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
-azd env set AZURE_PROVISION_VM true # Optional but recommended
-azd env set AZURE_VM_USERNAME myadminusername # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-username-requirements-when-creating-a-vm-
-azd env set AZURE_VM_PASSWORD mypassword # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-password-requirements-when-creating-a-vm-
-azd provision
-```
-4. Log into your new VM using [Azure Bastion](https://learn.microsoft.com/azure/bastion/tutorial-create-host-portal#connect). Validate the chat app is accessible from the virtual machine using a web browser.
+
+  ```shell
+  azd env set AZURE_USE_PRIVATE_ENDPOINT true
+  azd env set AZURE_PUBLIC_NETWORK_ACCESS Enabled
+  azd up
+  ```
+
+1. Validate that you can connect to the chat app and it's working as expected from the internet.
+1. Re-provision the app with public access disabled.
+
+  ```shell
+  azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
+  azd env set AZURE_PROVISION_VM true # Optional but recommended
+  azd env set AZURE_VM_USERNAME myadminusername # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-username-requirements-when-creating-a-vm-
+  azd env set AZURE_VM_PASSWORD mypassword # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-password-requirements-when-creating-a-vm-
+  azd provision
+  ```
+
+1. Log into your new VM using [Azure Bastion](https://learn.microsoft.com/azure/bastion/tutorial-create-host-portal#connect). Validate the chat app is accessible from the virtual machine using a web browser.