Changes for full vnet integration of data collected through block blo…

docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs.md

@@ -133,8 +133,7 @@ The following steps assume you have noted down the resource group name, storage
 :::
 
 * [Step 1: Authorize App Service read from storage account](#step-1-authorize-app-service-to-read-from-storage-account) - Enables the Azure functions to read from the storage account.
-* [Step 2: Create an Event Grid Subscription](#step-2-create-an-event-grid-subscription) - Subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-enabling-vnet-integration-optional) above.
-* [Step 3. Enabling Vnet Integration(Optional)](#step-3-enabling-vnet-integration-optional)
+* [Step 2: Create an Event Grid Subscription](#step-2-create-an-event-grid-subscription) - Subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-configure-azure-resources-using-arm-template) above.
 
 ### Step 1: Authorize App Service to read from storage account
 
@@ -157,61 +156,47 @@ To authorize the App Service to list the Storage Account key, do the following:
 
    * **Subscription**: Choose Pay as you Go.
    * **Managed Identity**: Choose Function App.
-   * **Select**:  **Select SUMOBRDLQProcessor\<unique_prefix\>** and **SUMORTaskConsumer\<unique_prefix\>** app services which are created by the ARM template. Click **Select**.
+   * **Select**:  **Select SUMOBRDLQProcessor\<unique_prefix\>** and **SUMOBRTaskConsumer\<unique_prefix\>** app services which are created by the ARM template. Click **Select**.
 1. Click **Review + assign**
 1. Click **Save**.
 
 ### Step 2: Create an Event Grid Subscription
 
-This section provides instructions for creating an event grid subscription, that subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-enabling-vnet-integration-optional) above.
+This section provides instructions for creating an event grid subscription, that subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-configure-azure-resources-using-arm-template) above.
 
 To create an event grid subscription, do the following:
 
-1. In the left pane of Azure portal click **All Services**, then search for and click **Event Grid Subscriptions**.
+1. Go to the storage account which needs to be monitored additionally. Go under Events blade in left pane.
 
-    ![AzureBlob_EventGridSubscriptions.png](/img/send-data/AzureBlob_EventGridSubscriptions.png)
-
-1. At the top of the **Event subscriptions** page, click **+Event Subscription**.
+1. At the top of the **Event subscriptions** tab, click **+Event Subscription** to create new event subscription.
 
     ![AzureBlob_EventSubscriptionsPage.png](/img/send-data/AzureBlob_EventSubscriptionsPage.png)
 
-    The Create Event Subscription dialog appears.
-
-    ![AzureBlob_CreatEventSubscription_dialog.png](/img/send-data/AzureBlob_CreatEventSubscription_dialog.png)
-
 1. Specify the following values for **Event Subscription Details**:
 
    * **Name:** Fill the event subscription name.
    * **Event Schema:** Select **Event Grid Schema**.
 
 1. Specify the following values for **Topic Details**:
 
-   * **Topic Type**. Select Storage Accounts.
-   * **Subscription**. Select Pay As You Go
-   * **Resource Group**. Select the Resource Group for the Storage Account to which your Azure service will export logs, from where you want to ingest logs.
-   * **Resource**. Select the Storage Account you configured, from where you want to ingest logs.
    * **System Topic Name**. Provide the topic name, if the system topic already exists then it will automatically select the existing topic.
-    :::note
-    If you do not see your configured Storage Account in the dropdown menu, make sure you met the requirements in [Requirements](#requirements) section.
-    :::
-
+
 1. Specify the following details for Event Types:
 
-   * Uncheck the **Subscribe to all event types** box.
-   * Select **Blob Created** from the **Define Event Types** dropdown.
+   * Select **Blob Created** from the **Filter to Event Types** dropdown.
 
 1. Specify the following details for Endpoint Types:
 
    * **Endpoint Type**. Select **Event Hubs** from the dropdown.
-   * **Endpoint.**  Click on **Select an endpoint.**
+   * **Endpoint.**  Click on **Configure an endpoint.**
 
     The Select Event Hub dialog appears.
 
     ![AzureBlob_SelectEventHub-EventGrid.png](/img/send-data/AzureBlob_SelectEventHub-EventGrid.png)
 
 1. Specify the following Select Event Hub parameters, then click **Confirm Selection.**
 
-   * **Resource Group**. Select the resource group you created [Step 3](#step-3-enabling-vnet-integration-optional) in which all the resources created by ARM template are present.
+   * **Resource Group**. Select the resource group you created [Step 3](#step-3-configure-azure-resources-using-arm-template) in which all the resources created by ARM template are present.
    * **Event Hub Namespace**. Select **SUMOBREventHubNamespace\<*unique string*\\>**.
    * **Event Hub**. Select **blobreadereventhub** from the dropdown.
 
@@ -226,9 +211,9 @@ To create an event grid subscription, do the following:
 
 1. Verify the deployment was successful by checking **Notifications** in the top right corner of the Azure Portal.
 
-### Step 3: Enabling VNet Integration (Optional)
+## Enabling VNet Integration (Optional)
 
-Assuming you have used the modified template which uses standard/premium plan for BlobTaskConsumer and [DLQTaskConsumer](https://portal.azure.com/#blade/WebsitesExtension/FunctionMenuBlade/resourceId/%2Fsubscriptions%2Fc088dc46-d692-42ad-a4b6-9a542d28ad2a%2FresourceGroups%2Fleast%2Fproviders%2FMicrosoft.Web%2Fsites%2FSUMOBRDLQProcessorekbxzlepnhs4g%2Ffunctions%2FDLQTaskConsumer) functions. This assumes that your storage account access is enabled for selected networks.
+This assumes that your storage account access is not public and is enabled for selected networks i.e. your storage account is behind a virtual network. This requires you to used the modified template which uses standard/premium plan for BlobTaskConsumer and DLQTaskConsumer functions. In case you want the whole data pipeline sending logs to sumo logic, to be under a virtual network follow the instruction [here](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/full-vnet-integration.md).
 
 1. Create a subnet in a virtual network using the instructions in the [doc](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-subnet#add-a-subnet). If you have multiple accounts in the same region you can skip step 2 below and use the same subnet and add it to the storage account as mentioned in step 3.
 1. Perform below steps for both BlobTaskConsumer and [DLQTaskConsumer](https://portal.azure.com/#blade/WebsitesExtension/FunctionMenuBlade/resourceId/%2Fsubscriptions%2Fc088dc46-d692-42ad-a4b6-9a542d28ad2a%2FresourceGroups%2Fleast%2Fproviders%2FMicrosoft.Web%2Fsites%2FSUMOBRDLQProcessorekbxzlepnhs4g%2Ffunctions%2FDLQTaskConsumer) function apps.
@@ -242,15 +227,10 @@ Assuming you have used the modified template which uses standard/premium plan fo
 
     ![azureblob-vnet](/img/send-data/azureblob-vnet.png)
 
-   1. Also copy the outbound ip addresses you’ll need to add it in firewall configuration of your storage account.
-
-    ![azureblob-outboundip](/img/send-data/azureblob-outboundip.png)
-
 1. Go to your storage account from where you want to collect logs from. Go to Networking and add the same Vnet and subnet.
 
     ![azureblob-storageacct](/img/send-data/azureblob-storageacct.png)
 
-1. Add the outbound ip addresses (copied in step 2.d) from both BlobTaskConsumer and [DLQTaskConsumer](https://portal.azure.com/#blade/WebsitesExtension/FunctionMenuBlade/resourceId/%2Fsubscriptions%2Fc088dc46-d692-42ad-a4b6-9a542d28ad2a%2FresourceGroups%2Fleast%2Fproviders%2FMicrosoft.Web%2Fsites%2FSUMOBRDLQProcessorekbxzlepnhs4g%2Ffunctions%2FDLQTaskConsumer) functions under Firewall with each ip in a single row of Address range column.
 1. Verify by going to the subnet. You should see Subnet delegation and service endpoints as shown in the screenshot below.
 
     ![azureblob-subnet](/img/send-data/azureblob-subnet.png)

docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/full-vnet-integration.md

@@ -0,0 +1,103 @@
+---
+id: full-vnet-integration
+title: Collect logs from Azure Blob Storage with full Virtual Network (VNet) Integration
+sidebar_label: Collect block blob with full Virtual Network integration
+description: Configure a pipeline to ship logs from the Azure Blob Storage throughout the Virtual Network and then to an HTTP source on a hosted collector in Sumo Logic.
+---
+
+import useBaseUrl from '@docusaurus/useBaseUrl';
+
+The current solution for ingesting block blob data from an Azure Storage Account into Sumo Logic sets up a pipeline that assumes public access is enabled on the storage account being monitored.
+If you prefer to restrict access and keep your storage account behind a firewall, refer to the instructions [here](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs/). However, if your security requirements demand that all Azure resources deployed via the ARM template, including the Storage Account, Event Hub, Azure Functions, and Service Bus, are fully integrated with a Virtual Network, follow the steps outlined below.
+
+1. Download the ARM template [https://github.com/SumoLogic/sumologic-azure-function/blob/azure\_premium\_template\_vnet\_integration/BlockBlobReader/src/blobreaderdeploywithPremiumPlan.json](https://github.com/SumoLogic/sumologic-azure-function/blob/azure_premium_template_vnet_integration/BlockBlobReader/src/blobreaderdeploywithPremiumPlan.json) that provisions the required resources, including a premium-tier Service Bus.
+2. Create the following networking resources:
+   * Virtual Network. For example, `brvnet`.
+       :::note
+       Only the Storage service endpoint associated with the functions and storage accounts is needed for the subnet.
+       :::
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-vnet-creation.png')} alt="Virtual Network creation with storage service endpoint" width="800" />
+   * Subnet. For example, `brsubnet`.
+   * Network Security Group (NSG). For example, `brnsg`.
+       :::note
+       NSG rules remain as default; no changes required.
+       :::
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-NSG-rules.png')} alt="NSG rules configuration" width="800" />
+3. Enable the Virtual Network integration on each function app by navigating to **Function App** > **Networking** > **Outbound Traffic Configuration**.  
+   <img src={useBaseUrl('/img/send-data/blockblob/block-blob-task-consumer-with-vnet-integration-outbound.png')} alt="TaskConsumer VNet integration outbound configuration" width="800" />
+   <img src={useBaseUrl('/img/send-data/blockblob/block-blob-vnet-in-task-consumer.png')} alt="VNet integration in TaskConsumer" width="800" />
+4. Follow the steps below to restrict access to the Storage Account storing NSG flow logs, so that only certain networks can access it:
+   1. Navigate to **Storage Account** > **Networking** > **Firewalls and virtual networks**.
+   2. Choose the selected networks.
+   3. Select the same subnet that was configured for **SUMOBRTaskConsumer** and **SUMOBRDLQProcessor** during Virtual Networ integration.
+       :::note
+       No IP address whitelisting is needed.
+       :::
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-sa-flow-logs-networking.png')} alt="Storage account flow logs networking configuration" width="800" />
+5. Follow the steps below to restrict access to the ARM-created storage account, so that only certain networks can access it:
+   1. Navigate to **Storage Account** > **Networking**.
+   2. Choose the selected networks and allow access from your subnet.   
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-arm-template-sa-networking.png')} alt="ARM template storage account networking configuration" width="800" />
+6. Configure the inbound restrictions on all three Azure Functions:
+   1. Navigate to **Function App** > **Networking** > **Inbound Traffic Configuration** > **Access Restrictions**.
+   2. Allow only the subnet you created in Step 2.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-task-consumer-with-vnet-integration-inbound.png')} alt="TaskConsumer VNet integration inbound configuration" width="800" />
+7. For each function app, enable the function access to the Storage Account created by the ARM template by following the steps below:
+   1. Navigate to **Function App** > **Networking** > **VNet Integration** > **Configuration Routing**.
+   2. Select **Content storage**.
+   3. Select **Outbound internet traffic** under **Application routing**.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-function-networking-config.png')} alt="Function networking configuration" width="800" />
+   4. Set `WEBSITE_CONTENTOVERVNET` to `1` in environment variables for each function.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-setting-env-variable-function.png')} alt="Setting environment variable in function" width="800" />
+8. Restrict access to **Service Bus** and **Event Hub** by following the steps below, so that only certain networks can access them:
+   1. Navigate to **Service** > **Networking**.
+   2. Set access to **Selected networks**, and select the previously created subnet in step 1.
+   3. Set **Allow trusted Microsoft services to bypass this firewall** to **Yes**.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-event-hub-networking.png')} alt="Event Hub networking configuration" width="800" />
+9. Secure the Event Grid with managed identity to allow Event Grid to publish to Event Hub:
+   1. Enable **System assigned** identity on the Event Grid Topic.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-system-assigned-identity-topic.png')} alt="System-assigned identity for topic" width="800" />
+   2. Assign the identity to the Azure Event Hubs Data Sender role on the Event Hub namespace under **Access Control (IAM)** > **Role Assignments**.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-event-hub-namespace-add-identity.png')} alt="Adding identity to Event Hub namespace" width="800" />
+   3. Configure the Event Grid subscription that uses an **Event Hub** as an endpoint and choose **System Assigned** identity for authentication.
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-event-hub-subscription-identity.png')} alt="Event Hub subscription identity configuration" width="800" />
+10. Ensure your Virtual Network has service endpoints enabled for:
+   - Storage
+   - Service Bus
+   - Event Hub
+       <img src={useBaseUrl('/img/send-data/blockblob/block-blob-service-endpoint-enabling-vnet.png')} alt="Enabling service endpoints in VNet" width="800" />
+11. To validate the function execution, navigate to **Function App** > **BlobTaskConsumer** > **Monitoring** > **Invocations**.
+    :::note
+    You should see the invocation logs if everything is correctly configured.
+    :::
+    <img src={useBaseUrl('/img/send-data/blockblob/block-blob-validation.png')} alt="Block blob validation logs" width="800" />
+12. Replace the standard Service Bus with a premium tier.
+     :::note
+     The Service Bus provisioned via the current ARM template is configured with the standard tier, which does not support Virtual Network integration. To enable Virtual Network integration, it is recommended to create a new Service Bus with the premium tier.
+     :::
+     Follow the steps below to create a new Service Bus on the premium tier:
+       1. Create a new premium Service Bus namespace:
+          1. Use the same resource group and location as the old Service Bus.
+          2. Enable partitioning.
+          3. Initially allow public access (can restrict later).
+       2. Create a new queue named `blobrangetaskqueue` with the following parameters:
+          1. Maximum queue size: 40 GB
+          2. Maximum message size: 1024 KB
+          3. Maximum delivery count: 3
+          4. Time to live: 14 days
+          5. Message lock duration: 5 minutes
+          6. Enable the dead letter queue.
+       3. Update the connection strings in all three functions (Producer, Consumer, DLQ):
+          Under **Shared access policies**, select the [RootManageSharedAccessKey](https://portal.azure.com/#) and copy the primary key from the newly created Service Bus on the premium tier as the value of `shared_access_key_value`:  
+          `Endpoint=sb://<servicebus_namespace_name>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<shared_access_key_value>`
+       4. Restrict Public Access:
+          1. Navigate to **Service Bus** > **Networking**.
+          2. Set **Public** network access to **Selected** networks.
+          3. Choose the subnet created earlier.
+
+### References
+
+- [https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-service-endpoints](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-service-endpoints)  
+- [https://learn.microsoft.com/en-us/azure/azure-functions/configure-networking-how-to?tabs=portal\#3-enable-application-and-configuration-routing](https://learn.microsoft.com/en-us/azure/azure-functions/configure-networking-how-to?tabs=portal#3-enable-application-and-configuration-routing)  
+- [https://learn.microsoft.com/en-us/azure/app-service/configure-vnet-integration-routing\#content-share](https://learn.microsoft.com/en-us/azure/app-service/configure-vnet-integration-routing#content-share)  
+- [https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings\#website\_contentovervnet](https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentovervnet)

docs/send-data/collect-from-other-data-sources/azure-monitoring/arm-integration-faq.md

@@ -371,4 +371,10 @@ To filter events by container name, do the following:
      ]
    }
    ```
+* Error in initiation of Azure functions created by ARM template with error message
+   ```System.Private.CoreLib: Access to the path 'C:\home\site\wwwroot' is denied```
+
+   This will also result unauthorized error in error logs for azure function. 
+   Every azure function always has a storage account associated with it for dumping logs, trigger event , metadata etc. Our arm template also creates 3 azure function and a single storage account (lets call it sumoBRlogs storage account). When this storage account access is restricted (not public) then this problem occurs. 
 
+   The solution is to do a virtual network (vnet) integration of azure function and allow the access to this virtual network to the sumoBRlogs storage account. Follow these [steps](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs/#enabling-vnet-integration-optional) to do a vnet integration. And set [this environment variable](https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings#website_contentovervnet) to 1, in all the three azure function created by ARM template - Producer, consumer and DLQ.

sidebars.ts

@@ -714,7 +714,8 @@ module.exports = {
                   collapsed: true,
                   link: { type: 'doc', id: 'send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/index' },
                   items: [
-                    'send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs'
+                    'send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs',
+                    'send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/full-vnet-integration',
                   ],
                 },
                 {