Optional deployment of Private Networks, Private Endpoints plus optional configuration of an ACL rule for all backend services (#864)

Add `AZURE_USE_PRIVATE_ENDPOINT` and `AZURE_PUBLIC_NETWORK_ACCESS` for private connectivity.
Add options for provisioning VM to connect to private chat app

Author: tonybaloney

docs/README.md

@@ -12,6 +12,7 @@ These are advanced topics that are not necessary for a basic deployment.
 * [Debugging the app on App Service](appservice.md)
 * [Local development](localdev.md)
 * [App customization](customization.md)
+* [Private access](private.md)
 * [Data ingestion](data_ingestion.md)
 * [Productionizing](productionizing.md)
 * [Alternative RAG chat samples](other_samples.md)

docs/deploy_private.md

@@ -0,0 +1,48 @@
+
+# Deploying with private access
+
+If you want to disable public access when deploying the Chat App, you can do so by setting `azd` environment values.
+
+## Before you begin
+
+Deploying with public access disabled adds additional cost to your deployment. Please see pricing for the following products:
+
+1. [Private Endpoints](https://azure.microsoft.com/pricing/details/private-link/)
+  1. The exact number of private endpoints created depends on the [optional features](./deploy_features.md) used.
+1. [Private DNS Zones](https://azure.microsoft.com/pricing/details/dns/)
+1. (Optional, but recommended)[Azure Virtual Machines](https://azure.microsoft.com/pricing/details/virtual-machines/windows/)
+1. (Optional, but recommended)[Azure Bastion](https://azure.microsoft.com/pricing/details/azure-bastion/)
+
+## Environment variables controlling private access
+
+1. `AZURE_PUBLIC_NETWORK_ACCESS`: Controls the value of public network access on supported Azure resources. Valid values are 'Enabled' or 'Disabled'.
+  1. When public network access is 'Enabled', Azure resources are open to the internet.
+  1. When public network access is 'Disabled', Azure resources are only accessible over a virtual network.
+1. `AZURE_USE_PRIVATE_ENDPOINT`: Controls deployment of [private endpoints](https://learn.microsoft.com/azure/private-link/private-endpoint-overview) which connect Azure resources to the virtual network.
+  1. When set to 'true', ensures private endpoints are deployed for connectivity even when `AZURE_PUBLIC_NETWORK_ACCESS` is 'Disabled'.
+  1. Note that private endpoints do not make the chat app accessible from the internet. Connections must be initiated from inside the virtual network.
+1. `AZURE_PROVISION_VM`: Controls deployment of a [virtual machine](https://learn.microsoft.com/azure/virtual-machines/overview) and [Azure Bastion](https://learn.microsoft.com/azure/bastion/bastion-overview). Azure Bastion allows you to securely connect to the virtual machine, without being connected virtual network. Since the virtual machine is connected to the virtual network, you are able to access the chat app.
+  1. You must set `AZURE_VM_USERNAME` and `AZURE_VM_PASSWORD` to provision the built-in administrator account with the virtual machine so you can log in through Azure Bastion.
+  1. By default, a server version of Windows is used for the VM. If you need to [enroll your device in Microsoft Intune](https://learn.microsoft.com/mem/intune/user-help/enroll-windows-10-device), you should use a desktop version of Windows by setting the following environment variables:
+    * `azd env set AZURE_VM_OS_PUBLISHER MicrosoftWindowsDesktop`
+    * `azd env set AZURE_VM_OS_OFFER Windows-11`
+    * `azd env set AZURE_VM_OS_VERSION win11-23h2-pro`
+
+## Recommended deployment strategy for private access
+
+1. Deploy the app with private endpoints enabled and public access enabled.
+```
+azd env set AZURE_USE_PRIVATE_ENDPOINT true
+azd env set AZURE_PUBLIC_NETWORK_ACCESS Enabled
+azd up
+```
+2. Validate that you can connect to the chat app and it's working as expected from the internet.
+3. Re-provision the app with public access disabled.
+```
+azd env set AZURE_PUBLIC_NETWORK_ACCESS Disabled
+azd env set AZURE_PROVISION_VM true # Optional but recommended
+azd env set AZURE_VM_USERNAME myadminusername # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-username-requirements-when-creating-a-vm-
+azd env set AZURE_VM_PASSWORD mypassword # https://learn.microsoft.com/azure/virtual-machines/windows/faq#what-are-the-password-requirements-when-creating-a-vm-
+azd provision
+```
+4. Log into your new VM using [Azure Bastion](https://learn.microsoft.com/azure/bastion/tutorial-create-host-portal#connect). Validate the chat app is accessible from the virtual machine using a web browser.

docs/login_and_acl.md

@@ -143,7 +143,7 @@ In both the chat and ask a question modes, under **Developer settings** optional
 
 The sample supports 2 main strategies for adding data with document level access control.
 
-* [Using the Add Documents API](#using-the-add-documents-api). Sample scripts are provided which use the Azure AI Search Service [Add Documents API](https://learn.microsoft.com/rest/api/searchservice/documents/?view=rest-searchservice-2023-11-01&tabs=HTTP) to directly manage access control information on _existing documents_ in the index.
+* [Using the Add Documents API](#using-the-add-documents-api). Sample scripts are provided which use the Azure AI Search Service Add Documents API to directly manage access control information on _existing documents_ in the index.
 * [Using prepdocs and Azure Data Lake Storage Gen 2](#azure-data-lake-storage-gen2-setup). Sample scripts are provided which set up an [Azure Data Lake Storage Gen 2](https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-introduction) account, set the [access control information](https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-access-control) on files and folders stored there, and ingest those documents into the search index with their  access control information.
 
 ### Using the Add Documents API

docs/productionizing.md

@@ -59,7 +59,7 @@ and scale up the maximum/minimum based on load.
 * **Authentication**: By default, the deployed app is publicly accessible.
   We recommend restricting access to authenticated users.
   See [Enabling authentication](./deploy_features.md#enabling-authentication) to learn how to enable authentication.
-* **Networking**: We recommend deploying inside a Virtual Network. If the app is only for
+* **Networking**: We recommend [deploying inside a Virtual Network](./deploy_private.md). If the app is only for
   internal enterprise use, use a private DNS zone. Also consider using Azure API Management (APIM)
   for firewalls and other forms of protection.
   For more details, read [Azure OpenAI Landing Zone reference architecture](https://techcommunity.microsoft.com/t5/azure-architecture-blog/azure-openai-landing-zone-reference-architecture/ba-p/3882102).

infra/abbreviations.json

@@ -105,6 +105,8 @@
   "operationalInsightsWorkspaces": "log-",
   "portalDashboards": "dash-",
   "powerBIDedicatedCapacities": "pbi-",
+  "privateEndpoint": "pe-",
+  "privateLink": "pl-",
   "purviewAccounts": "pview-",
   "recoveryServicesVaults": "rsv-",
   "resourcesResourceGroups": "rg-",
@@ -129,6 +131,7 @@
   "synapseWorkspacesSqlPoolsDedicated": "syndp",
   "synapseWorkspacesSqlPoolsSpark": "synsp",
   "timeSeriesInsightsEnvironments": "tsi-",
+  "virtualNetworks": "vnet-",
   "webServerFarms": "plan-",
   "webSitesAppService": "app-",
   "webSitesAppServiceEnvironment": "ase-",

infra/core/ai/cognitiveservices.bicep

@@ -13,24 +13,29 @@ param publicNetworkAccess string = 'Enabled'
 param sku object = {
   name: 'S0'
 }
+param ipRules array = []
+@allowed([ 'None', 'AzureServices' ])
+param bypass string = 'None'
 
-param allowedIpRules array = []
-param networkAcls object = empty(allowedIpRules) ? {
+var networkAcls = {
   defaultAction: 'Allow'
-} : {
-  ipRules: allowedIpRules
-  defaultAction: 'Deny'
 }
 
-resource account 'Microsoft.CognitiveServices/accounts@2023-05-01' = {
+var networkAclsWithBypass = {
+  defaultAction: 'Allow'
+  bypass: bypass
+}
+
+resource account 'Microsoft.CognitiveServices/accounts@2023-10-01-preview' = {
   name: name
   location: location
   tags: tags
   kind: kind
   properties: {
     customSubDomainName: customSubDomainName
     publicNetworkAccess: publicNetworkAccess
-    networkAcls: networkAcls
+    // Document Intelligence (FormRecognizer) does not support bypass in network acls
+    networkAcls: kind == 'FormRecognizer' ? networkAcls : networkAclsWithBypass
     disableLocalAuth: disableLocalAuth
   }
   sku: sku
@@ -52,4 +57,4 @@ resource deployment 'Microsoft.CognitiveServices/accounts/deployments@2023-05-01
 
 output endpoint string = account.properties.endpoint
 output id string = account.id
-output name string = account.name
+output name string = account.name

infra/core/host/appservice.bicep

@@ -8,6 +8,7 @@ param applicationInsightsName string = ''
 param appServicePlanId string
 param keyVaultName string = ''
 param managedIdentity bool = !empty(keyVaultName)
+param virtualNetworkSubnetId string = ''
 
 // Runtime Properties
 @allowed([
@@ -44,44 +45,54 @@ param serverAppId string = ''
 @secure()
 param clientSecretSettingName string = ''
 param authenticationIssuerUri string = ''
+@allowed([ 'Enabled', 'Disabled' ])
+param publicNetworkAccess string = 'Enabled'
 param enableUnauthenticatedAccess bool = false
 
 var msftAllowedOrigins = [ 'https://portal.azure.com', 'https://ms.portal.azure.com' ]
 var loginEndpoint = environment().authentication.loginEndpoint
 var loginEndpointFixed = lastIndexOf(loginEndpoint, '/') == length(loginEndpoint) - 1 ? substring(loginEndpoint, 0, length(loginEndpoint) - 1) : loginEndpoint
-var allMsftAllowedOrigins = !(empty(clientAppId)) ? union(msftAllowedOrigins, [loginEndpointFixed]) : msftAllowedOrigins
+var allMsftAllowedOrigins = !(empty(clientAppId)) ? union(msftAllowedOrigins, [ loginEndpointFixed ]) : msftAllowedOrigins
 
 // .default must be the 1st scope for On-Behalf-Of-Flow combined consent to work properly
 // Please see https://learn.microsoft.com/entra/identity-platform/v2-oauth2-on-behalf-of-flow#default-and-combined-consent
-var requiredScopes = ['api://${serverAppId}/.default', 'openid', 'profile', 'email', 'offline_access']
-var requiredAudiences = ['api://${serverAppId}']
+var requiredScopes = [ 'api://${serverAppId}/.default', 'openid', 'profile', 'email', 'offline_access' ]
+var requiredAudiences = [ 'api://${serverAppId}' ]
+
+var coreConfig = {
+  linuxFxVersion: linuxFxVersion
+  alwaysOn: alwaysOn
+  ftpsState: ftpsState
+  appCommandLine: appCommandLine
+  numberOfWorkers: numberOfWorkers != -1 ? numberOfWorkers : null
+  minimumElasticInstanceCount: minimumElasticInstanceCount != -1 ? minimumElasticInstanceCount : null
+  minTlsVersion: '1.2'
+  use32BitWorkerProcess: use32BitWorkerProcess
+  functionAppScaleLimit: functionAppScaleLimit != -1 ? functionAppScaleLimit : null
+  healthCheckPath: healthCheckPath
+  cors: {
+    allowedOrigins: union(allMsftAllowedOrigins, allowedOrigins)
+  }
+}
+
+var appServiceProperties = {
+  serverFarmId: appServicePlanId
+  siteConfig: coreConfig
+  clientAffinityEnabled: clientAffinityEnabled
+  httpsOnly: true
+  // Always route traffic through the vnet
+  // See https://learn.microsoft.com/azure/app-service/configure-vnet-integration-routing#configure-application-routing
+  vnetRouteAllEnabled: !empty(virtualNetworkSubnetId)
+  virtualNetworkSubnetId: !empty(virtualNetworkSubnetId) ? virtualNetworkSubnetId : null
+  publicNetworkAccess: publicNetworkAccess
+}
 
 resource appService 'Microsoft.Web/sites@2022-03-01' = {
   name: name
   location: location
   tags: tags
   kind: kind
-  properties: {
-    serverFarmId: appServicePlanId
-    siteConfig: {
-      linuxFxVersion: linuxFxVersion
-      alwaysOn: alwaysOn
-      ftpsState: ftpsState
-      minTlsVersion: '1.2'
-      appCommandLine: appCommandLine
-      numberOfWorkers: numberOfWorkers != -1 ? numberOfWorkers : null
-      minimumElasticInstanceCount: minimumElasticInstanceCount != -1 ? minimumElasticInstanceCount : null
-      use32BitWorkerProcess: use32BitWorkerProcess
-      functionAppScaleLimit: functionAppScaleLimit != -1 ? functionAppScaleLimit : null
-      healthCheckPath: healthCheckPath
-      cors: {
-        allowedOrigins: union(allMsftAllowedOrigins, allowedOrigins)
-      }
-    }
-    clientAffinityEnabled: clientAffinityEnabled
-    httpsOnly: true
-  }
-
+  properties: appServiceProperties
   identity: { type: managedIdentity ? 'SystemAssigned' : 'None' }
 
   resource configAppSettings 'config' = {
@@ -91,7 +102,7 @@ resource appService 'Microsoft.Web/sites@2022-03-01' = {
         SCM_DO_BUILD_DURING_DEPLOYMENT: string(scmDoBuildDuringDeployment)
         ENABLE_ORYX_BUILD: string(enableOryxBuild)
       },
-      runtimeName == 'python' ? { PYTHON_ENABLE_GUNICORN_MULTIWORKERS: 'true'} : {},
+      runtimeName == 'python' ? { PYTHON_ENABLE_GUNICORN_MULTIWORKERS: 'true' } : {},
       !empty(applicationInsightsName) ? { APPLICATIONINSIGHTS_CONNECTION_STRING: applicationInsights.properties.ConnectionString } : {},
       !empty(keyVaultName) ? { AZURE_KEY_VAULT_ENDPOINT: keyVault.properties.vaultUri } : {})
   }
@@ -140,7 +151,7 @@ resource appService 'Microsoft.Web/sites@2022-03-01' = {
             openIdIssuer: authenticationIssuerUri
           }
           login: {
-            loginParameters: ['scope=${join(union(requiredScopes, additionalScopes), ' ')}']
+            loginParameters: [ 'scope=${join(union(requiredScopes, additionalScopes), ' ')}' ]
           }
           validation: {
             allowedAudiences: union(requiredAudiences, additionalAllowedAudiences)
@@ -167,6 +178,7 @@ resource applicationInsights 'Microsoft.Insights/components@2020-02-02' existing
   name: applicationInsightsName
 }
 
+output id string = appService.id
 output identityPrincipalId string = managedIdentity ? appService.identity.principalId : ''
 output name string = appService.name
 output uri string = 'https://${appService.properties.defaultHostName}'

infra/core/host/vm.bicep

@@ -0,0 +1,68 @@
+param name string
+param location string
+param vmSize string = 'Standard_DS1_v2'
+param adminUsername string
+@secure()
+param adminPassword string
+param osVersion string = '2022-datacenter-azure-edition'
+param osPublisher string = 'MicrosoftWindowsServer'
+param osOffer string = 'WindowsServer'
+param nicId string
+param securityType string = 'TrustedLaunch'
+
+var securityProfileJson = {
+  uefiSettings: {
+    secureBootEnabled: true
+    vTpmEnabled: true
+  }
+  securityType: securityType
+}
+
+resource vm 'Microsoft.Compute/virtualMachines@2022-03-01' = {
+  name: name
+  location: location
+  properties: {
+    hardwareProfile: {
+      vmSize: vmSize
+    }
+    osProfile: {
+      computerName: name
+      adminUsername: adminUsername
+      adminPassword: adminPassword
+    }
+    storageProfile: {
+      imageReference: {
+        publisher: osPublisher
+        offer: osOffer
+        sku: osVersion
+        version: 'latest'
+      }
+      osDisk: {
+        createOption: 'FromImage'
+        managedDisk: {
+          storageAccountType: 'StandardSSD_LRS'
+        }
+      }
+      dataDisks: [
+        {
+          diskSizeGB: 1023
+          lun: 0
+          createOption: 'Empty'
+        }
+      ]
+    }
+    networkProfile: {
+      networkInterfaces: [
+        {
+          id: nicId
+        }
+      ]
+    }
+    diagnosticsProfile: {
+      bootDiagnostics: {
+        enabled: true
+      }
+    }
+    securityProfile: ((securityType == 'TrustedLaunch') ? securityProfileJson : null)
+  }
+}

infra/core/monitor/applicationinsights.bicep

@@ -4,6 +4,10 @@ param dashboardName string = ''
 param location string = resourceGroup().location
 param tags object = {}
 param logAnalyticsWorkspaceId string
+@allowed([ 'Enabled', 'Disabled' ])
+param publicNetworkAccessForIngestion string = 'Enabled'
+@allowed([ 'Enabled', 'Disabled' ])
+param publicNetworkAccessForQuery string = 'Enabled'
 
 resource applicationInsights 'Microsoft.Insights/components@2020-02-02' = {
   name: name
@@ -13,10 +17,12 @@ resource applicationInsights 'Microsoft.Insights/components@2020-02-02' = {
   properties: {
     Application_Type: 'web'
     WorkspaceResourceId: logAnalyticsWorkspaceId
+    publicNetworkAccessForIngestion: publicNetworkAccessForIngestion
+    publicNetworkAccessForQuery: publicNetworkAccessForQuery
   }
 }
 
-module applicationInsightsDashboard 'applicationinsights-dashboard.bicep' =  if (!empty(dashboardName)) {
+module applicationInsightsDashboard 'applicationinsights-dashboard.bicep' = if (!empty(dashboardName)) {
   name: 'application-insights-dashboard'
   params: {
     name: dashboardName
@@ -28,3 +34,4 @@ module applicationInsightsDashboard 'applicationinsights-dashboard.bicep' =  if
 output connectionString string = applicationInsights.properties.ConnectionString
 output instrumentationKey string = applicationInsights.properties.InstrumentationKey
 output name string = applicationInsights.name
+output id string = applicationInsights.id

infra/core/monitor/loganalytics.bicep

@@ -2,21 +2,27 @@ metadata description = 'Creates a Log Analytics workspace.'
 param name string
 param location string = resourceGroup().location
 param tags object = {}
+@allowed([ 'Enabled', 'Disabled' ])
+param publicNetworkAccessForIngestion string = 'Enabled'
+@allowed([ 'Enabled', 'Disabled' ])
+param publicNetworkAccessForQuery string = 'Enabled'
 
 resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2021-12-01-preview' = {
   name: name
   location: location
   tags: tags
-  properties: any({
+  properties: {
     retentionInDays: 30
     features: {
       searchVersion: 1
     }
     sku: {
       name: 'PerGB2018'
     }
-  })
+    publicNetworkAccessForIngestion: publicNetworkAccessForIngestion
+    publicNetworkAccessForQuery: publicNetworkAccessForQuery
+  }
 }
 
 output id string = logAnalytics.id
-output name string = logAnalytics.name
+output name string = logAnalytics.name