Skip to content

Commit 3f833ad

Browse files
wanlonghenrywanlonghenry
committed
add template for multi-tetant for multiple clusters support
1 parent 38f9096 commit 3f833ad

File tree

3 files changed

+410
-0
lines changed

3 files changed

+410
-0
lines changed

‎scripts/onboarding/aks/multi-tenancy-multiclusters/README.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Multi-Tenant Container Insights Setup with Shared DCR/DCE
2+
3+
This guide explains how to set up Container Insights monitoring for multiple AKS clusters using a shared Data Collection Rule (DCR) and Data Collection Endpoint (DCE).
4+
5+
## Setup Process
6+
7+
### 1. Deploy Shared DCR and DCE
8+
9+
First, deploy the shared DCR and DCE using the `existingClusterOnboarding.json` template. This creates the core monitoring infrastructure that will be shared across multiple clusters.
10+
11+
```bash
12+
az deployment group create \
13+
--name shared-monitoring-setup \
14+
--resource-group <resource-group-name> \
15+
--template-file existingClusterOnboarding.json \
16+
--parameters existingClusterParam.json
17+
```
18+
After deployment, note the DCR ID from the output. You'll need this to connect clusters.
19+
20+
### 2. Connect AKS Clusters to the Shared DCR
21+
22+
After deploying the DCR/DCE setup, you'll need to connect your AKS clusters. Below are simple commands to connect multiple clusters:
23+
24+
#### For Multiple Clusters (Bash)
25+
26+
```bash
27+
# Store the DCR ID
28+
DCR_ID="/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Insights/dataCollectionRules/<dcr-name>"
29+
30+
# If using private link, store the Config DCE ID
31+
CONFIG_DCE_ID="/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Insights/dataCollectionEndpoints/<dce-name>"
32+
33+
# Loop through your clusters
34+
for cluster in \
35+
"/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ContainerService/managedClusters/<cluster1-name>" \
36+
"/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ContainerService/managedClusters/<cluster2-name>"; do
37+
echo "Connecting cluster: $cluster"
38+
39+
# Create DCR association
40+
az monitor data-collection rule association create \
41+
--name "aks-dcr-association" \
42+
--rule-id "$DCR_ID" \
43+
--resource "$cluster"
44+
45+
# If using private link, create Config DCE association
46+
if [ ! -z "$CONFIG_DCE_ID" ]; then
47+
az monitor data-collection rule association create \
48+
--name "configurationAccessEndpoint" \
49+
--endpoint-id "$CONFIG_DCE_ID" \
50+
--resource "$cluster"
51+
fi
52+
done
53+
```
54+
55+
### Notes
56+
57+
1. The shared DCR/DCE setup needs to be done only once. After that, you can connect multiple clusters to the same DCR.
58+
59+
2. Each cluster needs:
60+
- DCR Association (always required)
61+
- Config DCE Association (only required when using private link)
62+
63+
3. Parameters explanation:
64+
- `subscriptionId`: The subscription where DCR/DCE will be created
65+
- `resourceGroupName`: Resource group for DCR/DCE resources
66+
- `location`: Location for the config DCE (only used with private link)
67+
- `workspaceRegion`: Region of the Log Analytics workspace
68+
- `workspaceResourceId`: Full resource ID of the Log Analytics workspace
69+
- `k8sNamespaces`: Array of Kubernetes namespaces to collect logs from
70+
- `transformKql`: Optional KQL query for log transformation
71+
- `useAzureMonitorPrivateLinkScope`: Whether to use private link scope
72+
- `azureMonitorPrivateLinkScopeResourceId`: Resource ID of private link scope (if used)
73+
- `resourceTagValues`: Tags to apply to the created resources
74+
75+
4. To list existing DCRAs for a cluster:
76+
```bash
77+
az monitor data-collection rule association list --resource "<aks-cluster-resource-id>"
78+
```
79+
80+
5. To remove a DCRA:
81+
```bash
82+
az monitor data-collection rule association delete \
83+
--name "aks-dcr-association" \
84+
--resource "<aks-cluster-resource-id>"
85+
```

‎scripts/onboarding/aks/multi-tenancy-multiclusters/existingClusterOnboarding.json‎

Lines changed: 281 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,281 @@
1+
{
2+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
3+
"contentVersion": "1.0.0.0",
4+
"parameters": {
5+
"resourceGroupName": {
6+
"type": "string",
7+
"metadata": {
8+
"description": "Resource group where DCR and DCE will be created"
9+
}
10+
},
11+
"location": {
12+
"type": "string",
13+
"metadata": {
14+
"description": "Location for the config DCE"
15+
}
16+
},
17+
"subscriptionId": {
18+
"type": "string",
19+
"metadata": {
20+
"description": "Subscription ID where DCR and DCE will be created"
21+
}
22+
},
23+
"workspaceRegion": {
24+
"type": "string",
25+
"metadata": {
26+
"description": "Workspace Region for data collection rule"
27+
}
28+
},
29+
"workspaceResourceId": {
30+
"type": "string",
31+
"metadata": {
32+
"description": "Full Resource ID of the log analytics workspace that will be used for data destination. For example /subscriptions/00000000-0000-0000-0000-0000-00000000/resourceGroups/ResourceGroupName/providers/Microsoft.operationalinsights/workspaces/ws_xyz"
33+
}
34+
},
35+
"resourceTagValues": {
36+
"type": "object",
37+
"metadata": {
38+
"description": "Tags to use on ContainerInsights and DataCollectionRule Resources"
39+
}
40+
},
41+
"k8sNamespaces": {
42+
"type": "array",
43+
"metadata": {
44+
"description": "An array of Kubernetes namespaces for Multi-tenancy logs filtering"
45+
}
46+
},
47+
"transformKql": {
48+
"type": "string",
49+
"metadata": {
50+
"description": "KQL filter for ingestion transformation"
51+
}
52+
},
53+
"useAzureMonitorPrivateLinkScope": {
54+
"type": "bool",
55+
"metadata": {
56+
"description": "Flag to indicate if Azure Monitor Private Link Scope should be used or not"
57+
}
58+
},
59+
"azureMonitorPrivateLinkScopeResourceId": {
60+
"type": "string",
61+
"metadata": {
62+
"description": "Specify the Resource Id of the Azure Monitor Private Link Scope."
63+
}
64+
}
65+
},
66+
"variables": {
67+
"workspaceName": "[split(parameters('workspaceResourceId'),'/')[8]]",
68+
"workspaceLocation": "[replace(parameters('workspaceRegion'),' ', '')]",
69+
"dcrNameFull": "[Concat('MSCI-multi-tenancy-shared', '-', variables('workspaceLocation'), '-', uniqueString(parameters('workspaceResourceId')))]",
70+
"dcrName": "[if(greater(length(variables('dcrNameFull')), 64), substring(variables('dcrNameFull'), 0, 64), variables('dcrNameFull'))]",
71+
"ingestionDCENameFull": "[Concat('MSCI-multi-tenancy-shared', '-', variables('workspaceLocation'), '-', uniqueString(parameters('workspaceResourceId')))]",
72+
"ingestionDCEName": "[if(greater(length(variables('ingestionDCENameFull')), 43), substring(variables('ingestionDCENameFull'), 0, 43), variables('ingestionDCENameFull'))]",
73+
"ingestionDCE": "[if(endsWith(variables('ingestionDCEName'), '-'), substring(variables('ingestionDCEName'), 0, 42), variables('ingestionDCEName'))]",
74+
"ingestionDataCollectionEndpointId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroupName'), 'Microsoft.Insights/dataCollectionEndpoints', variables('ingestionDCE'))]",
75+
"configDCENameFull": "[Concat('MSCI-config-shared', '-', parameters('location'), '-', uniqueString(parameters('workspaceResourceId')))]",
76+
"configDCEName": "[if(greater(length(variables('configDCENameFull')), 43), substring(variables('configDCENameFull'), 0, 43), variables('configDCENameFull'))]",
77+
"configDCE": "[if(endsWith(variables('configDCEName'), '-'), substring(variables('configDCEName'), 0, 42), variables('configDCEName'))]",
78+
"configDataCollectionEndpointId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroupName'), 'Microsoft.Insights/dataCollectionEndpoints', variables('configDCE'))]",
79+
"dataCollectionRuleId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroupName'), 'Microsoft.Insights/dataCollectionRules', variables('dcrName'))]",
80+
"privateLinkScopeName": "[split(parameters('azureMonitorPrivateLinkScopeResourceId'),'/')[8]]",
81+
"privateLinkScopeResourceGroup": "[split(parameters('azureMonitorPrivateLinkScopeResourceId'),'/')[4]]",
82+
"privateLinkScopeSubscriptionId": "[split(parameters('azureMonitorPrivateLinkScopeResourceId'),'/')[2]]"
83+
},
84+
"resources": [
85+
{
86+
"condition": "[parameters('useAzureMonitorPrivateLinkScope')]",
87+
"type": "Microsoft.Insights/dataCollectionEndpoints",
88+
"apiVersion": "2022-06-01",
89+
"name": "[variables('configDCE')]",
90+
"location": "[parameters('location')]",
91+
"tags": "[parameters('resourceTagValues')]",
92+
"kind": "Linux",
93+
"properties": {
94+
"networkAcls": {
95+
"publicNetworkAccess": "[if(parameters('useAzureMonitorPrivateLinkScope'), 'Disabled', 'Enabled')]"
96+
}
97+
}
98+
},
99+
{
100+
"type": "Microsoft.Insights/dataCollectionEndpoints",
101+
"apiVersion": "2023-03-11",
102+
"name": "[variables('ingestionDCE')]",
103+
"location": "[variables('workspaceLocation')]",
104+
"tags": "[parameters('resourceTagValues')]",
105+
"kind": "Linux",
106+
"properties": {
107+
"networkAcls": {
108+
"publicNetworkAccess": "[if(parameters('useAzureMonitorPrivateLinkScope'), 'Disabled', 'Enabled')]"
109+
}
110+
}
111+
},
112+
{
113+
"type": "Microsoft.Resources/deployments",
114+
"name": "[Concat('shared-monitoring-dcr', '-', uniqueString(variables('dcrName')))]",
115+
"apiVersion": "2017-05-10",
116+
"subscriptionId": "[parameters('subscriptionId')]",
117+
"resourceGroup": "[parameters('resourceGroupName')]",
118+
"dependsOn": [
119+
"[variables('ingestionDCE')]"
120+
],
121+
"properties": {
122+
"mode": "Incremental",
123+
"template": {
124+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
125+
"contentVersion": "1.0.0.0",
126+
"parameters": {},
127+
"variables": {},
128+
"resources": [
129+
{
130+
"type": "Microsoft.Insights/dataCollectionRules",
131+
"apiVersion": "2023-03-11",
132+
"name": "[variables('dcrName')]",
133+
"location": "[parameters('workspaceRegion')]",
134+
"tags": "[parameters('resourceTagValues')]",
135+
"kind": "Linux",
136+
"properties": {
137+
"dataSources": {
138+
"extensions": [
139+
{
140+
"name": "ContainerLogV2Extension",
141+
"streams": [
142+
"Microsoft-ContainerLogV2-HighScale"
143+
],
144+
"extensionSettings": {
145+
"dataCollectionSettings": {
146+
"namespaces": "[parameters('k8sNamespaces')]"
147+
}
148+
},
149+
"extensionName": "ContainerLogV2Extension"
150+
}
151+
]
152+
},
153+
"destinations": {
154+
"logAnalytics": [
155+
{
156+
"workspaceResourceId": "[parameters('workspaceResourceId')]",
157+
"name": "ciworkspace"
158+
}
159+
]
160+
},
161+
"dataFlows": [
162+
{
163+
"streams": [
164+
"Microsoft-ContainerLogV2-HighScale"
165+
],
166+
"destinations": [
167+
"ciworkspace"
168+
],
169+
"transformKql": "[if(empty(parameters('transformKql')), json('null'), parameters('transformKql'))]"
170+
}
171+
],
172+
"dataCollectionEndpointId": "[variables('ingestionDataCollectionEndpointId')]"
173+
}
174+
}
175+
]
176+
},
177+
"parameters": {}
178+
}
179+
},
180+
{
181+
"condition": "[parameters('useAzureMonitorPrivateLinkScope')]",
182+
"type": "Microsoft.Resources/deployments",
183+
"name": "[Concat('shared-monitoring-ampls-scope-config', '-', uniqueString(parameters('workspaceResourceId')))]",
184+
"apiVersion": "2017-05-10",
185+
"subscriptionId": "[variables('privateLinkScopeSubscriptionId')]",
186+
"resourceGroup": "[variables('privateLinkScopeResourceGroup')]",
187+
"dependsOn": [
188+
"[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('configDCE'))]"
189+
],
190+
"properties": {
191+
"mode": "Incremental",
192+
"template": {
193+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
194+
"contentVersion": "1.0.0.0",
195+
"parameters": {},
196+
"variables": {},
197+
"resources": [
198+
{
199+
"type": "microsoft.insights/privatelinkscopes/scopedresources",
200+
"name": "[concat(variables('privateLinkScopeName'), '/', concat(variables('configDCE'), '-connection'))]",
201+
"apiVersion": "2021-07-01-preview",
202+
"properties": {
203+
"linkedResourceId": "[variables('configDataCollectionEndpointId')]"
204+
}
205+
}
206+
]
207+
},
208+
"parameters": {}
209+
}
210+
},
211+
{
212+
"condition": "[parameters('useAzureMonitorPrivateLinkScope')]",
213+
"type": "Microsoft.Resources/deployments",
214+
"name": "[Concat('shared-monitoring-ampls-scope-ingest', '-', uniqueString(parameters('workspaceResourceId')))]",
215+
"apiVersion": "2017-05-10",
216+
"subscriptionId": "[variables('privateLinkScopeSubscriptionId')]",
217+
"resourceGroup": "[variables('privateLinkScopeResourceGroup')]",
218+
"dependsOn": [
219+
"[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('ingestionDCE'))]"
220+
],
221+
"properties": {
222+
"mode": "Incremental",
223+
"template": {
224+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
225+
"contentVersion": "1.0.0.0",
226+
"parameters": {},
227+
"variables": {},
228+
"resources": [
229+
{
230+
"type": "microsoft.insights/privatelinkscopes/scopedresources",
231+
"name": "[concat(variables('privateLinkScopeName'), '/', concat(variables('ingestionDCE'), '-connection'))]",
232+
"apiVersion": "2021-07-01-preview",
233+
"properties": {
234+
"linkedResourceId": "[variables('ingestionDataCollectionEndpointId')]"
235+
}
236+
}
237+
]
238+
},
239+
"parameters": {}
240+
}
241+
},
242+
{
243+
"condition": "[parameters('useAzureMonitorPrivateLinkScope')]",
244+
"type": "Microsoft.Resources/deployments",
245+
"name": "[Concat('shared-monitoring-ampls-scope', '-', uniqueString(parameters('workspaceResourceId')))]",
246+
"apiVersion": "2017-05-10",
247+
"subscriptionId": "[variables('privateLinkScopeSubscriptionId')]",
248+
"resourceGroup": "[variables('privateLinkScopeResourceGroup')]",
249+
"properties": {
250+
"mode": "Incremental",
251+
"template": {
252+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
253+
"contentVersion": "1.0.0.0",
254+
"parameters": {},
255+
"variables": {},
256+
"resources": [
257+
{
258+
"type": "microsoft.insights/privatelinkscopes/scopedresources",
259+
"name": "[concat(variables('privateLinkScopeName'), '/', concat(variables('workspaceName'), '-connection'))]",
260+
"apiVersion": "2021-07-01-preview",
261+
"properties": {
262+
"linkedResourceId": "[parameters('workspaceResourceId')]"
263+
}
264+
}
265+
]
266+
},
267+
"parameters": {}
268+
}
269+
}
270+
],
271+
"outputs": {
272+
"configDceId": {
273+
"type": "string",
274+
"value": "[variables('configDataCollectionEndpointId')]"
275+
},
276+
"dcrId": {
277+
"type": "string",
278+
"value": "[variables('dataCollectionRuleId')]"
279+
}
280+
}
281+
}

0 commit comments

Comments
 (0)