Merge pull request #225300 from Sohamdg081992/UpdateMultiDcrDoc

Update doc for sending Prometheus metrics to multiple workspace

Author: JamesJBarnett

articles/azure-monitor/essentials/prometheus-metrics-multiple-workspaces.md

@@ -13,7 +13,7 @@ Routing metrics to more Azure Monitor Workspaces can be done through the creatio
 
 ## Send same metrics to multiple Azure Monitor workspaces
 
-You can create multiple Data Collection Rules that point to the same Data Collection Endpoint for metrics to be sent to additional Azure Monitor Workspaces from the same Kubernetes cluster. Currently, this is only available through onboarding through Resource Manager templates. You can follow the [regular onboarding process](prometheus-metrics-enable.md) and then edit the same Resource Manager templates to add additional DCRs for your additional Azure Monitor Workspaces. You'll need to edit the template to add an additional parameters for every additional Azure Monitor workspace, add another DCR for every additional Azure Monitor workspace, and add an additional Azure Monitor workspace integration for Grafana.
+You can create multiple Data Collection Rules that point to the same Data Collection Endpoint for metrics to be sent to additional Azure Monitor Workspaces from the same Kubernetes cluster. In case you have a very high volume of metrics, a new Data Collection Endpoint can be created as well. Please refer to the service limits [document](../service-limits.md) regarding ingestion limits. Currently, this is only available through onboarding through Resource Manager templates. You can follow the [regular onboarding process](prometheus-metrics-enable.md) and then edit the same Resource Manager templates to add additional DCRs and DCEs(if applicable) for your additional Azure Monitor Workspaces. You'll need to edit the template to add an additional parameters for every additional Azure Monitor workspace, add another DCR for every additional Azure Monitor workspace, add another DCE (if applicable), add the Monitor Reader Role for the new Azure Monitor Workspace and add an additional Azure Monitor workspace integration for Grafana.
 
 - Add the following parameters:
   ```json
@@ -43,7 +43,18 @@ You can create multiple Data Collection Rules that point to the same Data Collec
   }
   ```
 
-- Add an additional DCR with the same Data Collection Endpoint. You *must* replace `<dcrName>`:
+- For high metric volume, add an additional Data Collection Endpoint. You *must* replace `<dceName>`:
+  ```json
+    {
+      "type": "Microsoft.Insights/dataCollectionEndpoints",
+      "apiVersion": "2021-09-01-preview",
+      "name": "[variables('dceName')]",
+      "location": "[parameters('azureMonitorWorkspaceLocation2')]",
+      "kind": "Linux",
+      "properties": {}
+    }
+  ```
+- Add an additional DCR with the same or a different Data Collection Endpoint. You *must* replace `<dcrName>`:
   ```json
   {
     "type": "Microsoft.Insights/dataCollectionRules",
@@ -84,7 +95,42 @@ You can create multiple Data Collection Rules that point to the same Data Collec
   }
   ```
 
-
+- Add an additional DCRA with the relevant Data Collection Rule. You *must* replace `<dcraName>`:
+     ```json
+    {
+      "type": "Microsoft.Resources/deployments",
+      "name": "<dcraName>",
+      "apiVersion": "2017-05-10",
+      "subscriptionId": "[variables('clusterSubscriptionId')]",
+      "resourceGroup": "[variables('clusterResourceGroup')]",
+      "dependsOn": [
+        "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
+        "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
+      ],
+      "properties": {
+        "mode": "Incremental",
+        "template": {
+          "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
+          "contentVersion": "1.0.0.0",
+          "parameters": {},
+          "variables": {},
+          "resources": [
+            {
+              "type": "Microsoft.ContainerService/managedClusters/providers/dataCollectionRuleAssociations",
+              "name": "[concat(variables('clusterName'),'/microsoft.insights/', variables('dcraName'))]",
+              "apiVersion": "2021-09-01-preview",
+              "location": "[parameters('clusterLocation')]",
+              "properties": {
+                "description": "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster.",
+                "dataCollectionRuleId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
+              }
+            }
+          ]
+        },
+        "parameters": {}
+      }
+    }
+    ```
 - Add an additional Grafana integration:
   ```json
   {
@@ -116,8 +162,21 @@ You can create multiple Data Collection Rules that point to the same Data Collec
         }
       }
   ```
-  Similar to the regular Resource Manager onboarding process, the `Monitoring Data Reader` role will need to be assigned for every Azure Monitor workspace linked to Grafana. This will allow the Azure Managed Grafana resource to read data from the Azure Monitor workspace and is a requirement for viewing the metrics.
+  - Assign `Monitoring Data Reader` role to read data from the new Azure Monitor Workspace:
 
+  ```json
+  {
+    "type": "Microsoft.Authorization/roleAssignments",
+    "apiVersion": "2022-04-01",
+    "name": "[parameters('roleNameGuid')]",
+    "scope": "[parameters('azureMonitorWorkspaceResourceId2')]",
+    "properties": {
+        "roleDefinitionId": "[concat('/subscriptions/', variables('clusterSubscriptionId'), '/providers/Microsoft.Authorization/roleDefinitions/', 'b0d8363b-8ddd-447d-831f-62ca05bff136')]",
+        "principalId": "[reference(resourceId('Microsoft.Dashboard/grafana', split(parameters('grafanaResourceId'),'/')[8]), '2022-08-01', 'Full').identity.principalId]"
+    }
+  }
+
+  ```
 ## Send different metrics to different Azure Monitor workspaces
 
 If you want to send some metrics to one Azure Monitor Workspace and other metrics to a different one, follow the above steps to add additional DCRs. The value of `microsoft_metrics_include_label` under the `labelIncludeFilter` in the DCR is the identifier for the workspace. To then configure which metrics are routed to which workspace, you can add an extra pre-defined label, `microsoft_metrics_account` to the metrics. The value should be the same as the corresponding `microsoft_metrics_include_label` in the DCR for that workspace. To add the label to the metrics, you can utilize `relabel_configs` in your scrape config. To send all metrics from one job to a certain workspace, add the following relabel config: