Skip to content

Commit 458b098

Browse files
TedA-MTedA-M
authored
Merge pull request #94789 from orspod/2019-11-follower
2019 11 follower
2 parents 3e8f39d + a1f4c62 commit 458b098

File tree

5 files changed

+270
-0
lines changed

5 files changed

+270
-0
lines changed

‎articles/data-explorer/follower.md

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
---
2+
title: Use follower database feature to attach databases in Azure Data Explorer
3+
description: Learn about how to attach databases in Azure Data Explorer using the follower database feature.
4+
author: orspod
5+
ms.author: orspodek
6+
ms.reviewer: gabilehner
7+
ms.service: data-explorer
8+
ms.topic: conceptual
9+
ms.date: 11/07/2019
10+
---
11+
12+
# Use follower database to attach databases in Azure Data Explorer
13+
14+
The **follower database** feature allows you to attach a database located in a different cluster to your Azure Data Explorer cluster. The **follower database** is attached in *read-only* mode, making it possible to view the data and run queries on the data that was ingested into the **leader database**. The follower database synchronizes changes in the leader databases. Due to the synchronization, there's a data lag of a few seconds to a few minutes in data availability. The length of the time lag depends on the overall size of the leader database metadata. The leader and follower databases use the same storage account to fetch the data. The storage is owned by the leader database. The follower database views the data without needing to ingest it. Since the attached database is a read-only database, the data, tables, and policies in the database can't be modified except for [caching policy](#configure-caching-policy), [principals](#manage-principals), and [permissions](#manage-permissions). Attached databases can't be deleted. They must be detached by the leader or follower and only then they can be deleted.
15+
16+
Attaching a database to a different cluster using the follower capability is used as the infrastructure to share data between organizations and teams. The feature is useful to segregate compute resources to protect a production environment from non-production use cases. Follower can also be used to associate the cost of Azure Data Explorer cluster to the party that runs queries on the data.
17+
18+
## Which databases are followed?
19+
20+
* A cluster can follow one database, several databases, or all databases of a leader cluster.
21+
* A single cluster can follow databases from multiple leader clusters.
22+
* A cluster can contain both follower databases and leader databases
23+
24+
## Prerequisites
25+
26+
1. If you don't have an Azure subscription, [create a free account](https://azure.microsoft.com/free/) before you begin.
27+
1. [Create cluster and DB](/azure/data-explorer/create-cluster-database-portal) for the leader and follower.
28+
1. [Ingest data](/azure/data-explorer/ingest-sample-data) to leader database using one of various methods discussed in [ingestion overview](/azure/data-explorer/ingest-data-overview).
29+
30+
## Attach a database
31+
32+
There are various methods you can use to attach a database. In this article, we discuss attaching a database using C# or an Azure Resource Manager template. To attach a database, you must have permissions on the leader cluster and the follower cluster. For more information about permissions, see [manage permissions](#manage-permissions).
33+
34+
### Attach a database using C#
35+
36+
**Needed NuGets**
37+
38+
* Install [Microsoft.Azure.Management.kusto](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
39+
* Install [Microsoft.Rest.ClientRuntime.Azure.Authentication for authentication](https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure.Authentication).
40+
41+
42+
```Csharp
43+
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
44+
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
45+
var clientSecret = "xxxxxxxxxxxxxx";//Client secret
46+
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
47+
48+
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
49+
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
50+
51+
var leaderResourceGroupName = "testrg";
52+
var followerResourceGroupName = "followerResouceGroup";
53+
var leaderClusterName = "leader";
54+
var followerClusterName = "follower";
55+
var attachedDatabaseConfigurationName = "adc";
56+
var databaseName = "db" // Can be specific database name or * for all databases
57+
var defaultPrincipalsModificationKind = "Union";
58+
var location = "North Central US";
59+
60+
AttachedDatabaseConfiguration attachedDatabaseConfigurationProperties = new AttachedDatabaseConfiguration()
61+
{
62+
ClusterResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{followerResourceGroupName}/providers/Microsoft.Kusto/Clusters/{followerClusterName}",
63+
DatabaseName = databaseName,
64+
DefaultPrincipalsModificationKind = defaultPrincipalsModificationKind,
65+
Location = location
66+
};
67+
68+
var attachedDatabaseConfigurations = resourceManagementClient.AttachedDatabaseConfigurations.CreateOrUpdate(followerResourceGroupName, followerClusterName, attachedDatabaseConfigurationName, attachedDatabaseConfigurationProperties);
69+
```
70+
71+
### Attach a database using an Azure Resource Manager template
72+
73+
In this section, you learn how to attach a database by using an [Azure Resource Manager template](../azure-resource-manager/resource-group-overview.md).
74+
75+
```json
76+
{
77+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
78+
"contentVersion": "1.0.0.0",
79+
"parameters": {
80+
"followerClusterName": {
81+
"type": "string",
82+
"defaultValue": "",
83+
"metadata": {
84+
"description": "Name of the follower cluster."
85+
}
86+
},
87+
"attachedDatabaseConfigurationsName": {
88+
"type": "string",
89+
"defaultValue": "",
90+
"metadata": {
91+
"description": "Name of the attached database configurations to create."
92+
}
93+
},
94+
"databaseName": {
95+
"type": "string",
96+
"defaultValue": "",
97+
"metadata": {
98+
"description": "The name of the database to follow. You can follow all databases by using '*'."
99+
}
100+
},
101+
"leaderClusterResourceId": {
102+
"type": "string",
103+
"defaultValue": "",
104+
"metadata": {
105+
"description": "Name of the leader cluster to create."
106+
}
107+
},
108+
"defaultPrincipalsModificationKind": {
109+
"type": "string",
110+
"defaultValue": "",
111+
"metadata": {
112+
"description": "The default principal modification kind."
113+
}
114+
},
115+
"location": {
116+
"type": "string",
117+
"defaultValue": "",
118+
"metadata": {
119+
"description": "Location for all resources."
120+
}
121+
}
122+
},
123+
"variables": {},
124+
"resources": [
125+
{
126+
"name": "[parameters('followerClusterName')]",
127+
"type": "Microsoft.Kusto/clusters",
128+
"sku": {
129+
"name": "Standard_D13_v2",
130+
"tier": "Standard",
131+
"capacity": 2
132+
},
133+
"apiVersion": "2019-09-07",
134+
"location": "[parameters('location')]"
135+
},
136+
{
137+
"name": "[concat(parameters('followerClusterName'), '/', parameters('attachedDatabaseConfigurationsName'))]",
138+
"type": "Microsoft.Kusto/clusters/attachedDatabaseConfigurations",
139+
"apiVersion": "2019-09-07",
140+
"location": "[parameters('location')]",
141+
"dependsOn": [
142+
"[resourceId('Microsoft.Kusto/clusters', parameters('followerClusterName'))]"
143+
],
144+
"properties": {
145+
"databaseName": "[parameters('databaseName')]",
146+
"clusterResourceId": "[parameters('leaderClusterResourceId')]",
147+
"defaultPrincipalsModificationKind": "[parameters('defaultPrincipalsModificationKind')]"
148+
}
149+
}
150+
]
151+
}
152+
```
153+
154+
### Deploy the template
155+
156+
You can deploy the Azure Resource Manager template by [using the Azure portal](https://portal.azure.com) or using powershell.
157+
158+
![template deployment](media/follower/template-deployment.png)
159+
160+
161+
|**Setting** |**Description** |
162+
|---------|---------|
163+
|Follower Cluster Name | The name of the follower cluster |
164+
|Attached Database Configurations Name | The name of the attached database configurations object. The name must be unique at the cluster level. |
165+
|Database Name | The name of the database to be followed. If you want to follow all the leader's databases, use '*'. |
166+
|Leader Cluster Resource ID | The resource ID of the leader cluster. |
167+
|Default Principals Modification Kind | The default principal modification kind. Can be `Union`, `Replace` or `None`. For more information about default principal modification kind, see [principal modification kind control command](/azure/kusto/management/cluster-follower?branch=master#alter-follower-database-principals-modification-kind). |
168+
|Location | The location of all the resources. The leader and the follower must be in the same location. |
169+
170+
### Verify that the database was successfully attached
171+
172+
To verify that the database was successfully attached, find your attached databases in the [Azure portal](https://portal.azure.com).
173+
174+
1. Navigate to the follower cluster and select **Databases**
175+
1. Search for new Read-only databases in the database list.
176+
177+
![Read-only follower database](media/follower/read-only-follower-database.png)
178+
179+
Alternatively:
180+
181+
1. Navigate to the leader cluster and select **Databases**
182+
2. Check that the relevant databases are marked as **SHARED WITH OTHERS** > **Yes**
183+
184+
![Read and write attached databases](media/follower/read-write-databases-shared.png)
185+
186+
## Detach the follower database using C#
187+
188+
### Detach the attached follower database from the follower cluster
189+
190+
Follower cluster can detach any attached database as follows:
191+
192+
```csharp
193+
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
194+
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
195+
var clientSecret = "xxxxxxxxxxxxxx";//Client secret
196+
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
197+
198+
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
199+
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
200+
201+
var followerResourceGroupName = "testrg";
202+
//The cluster and database that are created as part of the prerequisites
203+
var followerClusterName = "follower";
204+
var attachedDatabaseConfigurationsName = "adc";
205+
206+
resourceManagementClient.AttachedDatabaseConfigurations.Delete(followerResourceGroupName, followerClusterName, attachedDatabaseConfigurationsName);
207+
```
208+
209+
### Detach the attached follower database from the leader cluster
210+
211+
The leader cluster can detach any attached database as follows:
212+
213+
```csharp
214+
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
215+
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
216+
var clientSecret = "xxxxxxxxxxxxxx";//Client secret
217+
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
218+
219+
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
220+
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
221+
222+
var leaderResourceGroupName = "testrg";
223+
var followerResourceGroupName = "followerResouceGroup";
224+
var leaderClusterName = "leader";
225+
var followerClusterName = "follower";
226+
//The cluster and database that are created as part of the Prerequisites
227+
var followerDatabaseDefinition = new FollowerDatabaseDefinition()
228+
{
229+
AttachedDatabaseConfigurationName = "adc",
230+
ClusterResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{followerResourceGroupName}/providers/Microsoft.Kusto/Clusters/{followerClusterName}"
231+
};
232+
233+
resourceManagementClient.Clusters.DetachFollowerDatabases(leaderResourceGroupName, leaderClusterName, followerDatabaseDefinition);
234+
```
235+
236+
## Manage principals, permissions, and caching policy
237+
238+
### Manage principals
239+
240+
When attaching a database, specify the **default principals modification kind"**. The default is keeping the leader database collection of [authorized principals](/azure/kusto/management/access-control/index.md#authorization)
241+
242+
|**Kind** |**Description** |
243+
|---------|---------|
244+
|**Union** | The attached database principals will always include the original database principals plus additional new principals added to the follower database. |
245+
|**Replace** | No inheritance of principals from the original database. New principals must be created for the attached database. At least one principal needs to be added to block principal inheritance. |
246+
|**None** | The attached database principals include only the principals of the original database with no additional principals. |
247+
248+
For more information about using control commands to configure the authorized principals, see [Control commands for managing a follower cluster](/azure/kusto/management/cluster-follower.md).
249+
250+
### Manage permissions
251+
252+
Managing read-only database permission is the same as for all database types. See [manage permissions in the Azure portal](/azure/data-explorer/manage-database-permissions#manage-permissions-in-the-azure-portal).
253+
254+
### Configure caching policy
255+
256+
The follower database administrator can modify the [caching policy](/azure/kusto/management/cache-policy) of the attached database or any of its tables on the hosting cluster. The default is keeping the leader database collection of database and table-level caching policies. You can, for example, have a 30 day caching policy on the leader database for running monthly reporting and a three day caching policy on the follower database to query only the recent data for troubleshooting. For more information about using control commands to configure the caching policy on the follower database or table, see [Control commands for managing a follower cluster](/azure/kusto/management/cluster-follower.md).
257+
258+
## Limitations
259+
260+
* The follower and the leader clusters must be in the same region.
261+
* [Streaming ingestion](/azure/data-explorer/ingest-data-streaming) can't be used on a database that is being followed.
262+
* You can't delete a database that is attached to a different cluster before detaching it.
263+
* You can't delete a cluster that has a database attached to a different cluster before detaching it.
264+
* You can't stop a cluster that has attached follower or leader database(s).
265+
266+
## Next steps
267+
268+
* For information about follower cluster configuration, see [Control commands for managing a follower cluster](/azure/kusto/management/cluster-follower.md).

‎articles/data-explorer/media/follower/read-only-follower-database.png

87.3 KB
Loading

‎articles/data-explorer/media/follower/read-write-databases-shared.png

87.2 KB
Loading

‎articles/data-explorer/media/follower/template-deployment.png

26.2 KB
Loading

‎articles/data-explorer/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,8 @@
150150
href: check-cluster-health.md
151151
- name: Database
152152
items:
153+
- name: Follower databases
154+
href: follower.md
153155
- name: Manage database permissions
154156
href: manage-database-permissions.md
155157
- name: Policies

0 commit comments

Comments
 (0)