Skip to content

Commit b54443b

Browse files
lugoldbelugoldbe
lugoldbe
authored and
lugoldbe
committed
add an end to end example
1 parent b9fba5c commit b54443b

7 files changed

+574
-5
lines changed

‎articles/data-explorer/create-cluster-database-csharp.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@ Azure Data Explorer is a fast, fully managed data analytics service for real-tim
2929

3030
## Install C# Nuget
3131

32-
1. Install the [Azure Data Explorer (Kusto) nuget package](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
32+
* Install the [Azure Data Explorer (Kusto) nuget package](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
3333

34-
1. Install the [Microsoft.IdentityModel.Clients.ActiveDirectory nuget package](https://www.nuget.org/packages/Microsoft.IdentityModel.Clients.ActiveDirectory/) for authentication.
34+
* Install the [Microsoft.IdentityModel.Clients.ActiveDirectory nuget package](https://www.nuget.org/packages/Microsoft.IdentityModel.Clients.ActiveDirectory/) for authentication.
3535

3636
## Authentication
3737
For running the examples in this article, we need an Azure AD Application and service principal that can access resources. Check [create an Azure AD application](https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal) to create a free Azure AD Application and add role assignment at the subscription scope. It also shows how to get the `Directory (tenant) ID`, `Application ID`, and `Client Secret`.

‎articles/data-explorer/data-connection-event-grid-python.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ poller = kusto_management_client.data_connections.create_or_update(resource_grou
8181
|**Setting** | **Suggested value** | **Field description**|
8282
|---|---|---|
8383
| tenant_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | Your tenant ID. Also known as directory ID.|
84-
| subscriptionId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
84+
| subscription_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
8585
| client_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The client ID of the application that can access resources in your tenant.|
8686
| client_secret | *xxxxxxxxxxxxxx* | The client secret of the application that can access resources in your tenant. |
8787
| resource_group_name | *testrg* | The name of the resource group containing your cluster.|

‎articles/data-explorer/end-to-end-csharp.md

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
---
2+
title: 'Create an End to End example for Azure Data Explorer by using Python'
3+
description: In this article, you learn how to use Azure Data Explorer with an End to End example using Python.
4+
author: lucygoldbergmicrosoft
5+
ms.author: lugoldbe
6+
ms.reviewer: orspodek
7+
ms.service: data-explorer
8+
ms.topic: conceptual
9+
ms.date: 09/24/2019
10+
---
11+
12+
# A End-to-End example for ingesting blobs into Azure Data Explorer using Python
13+
14+
> [!div class="op_single_selector"]
15+
> * [C#](data-connection-csharp.md)
16+
> * [Python](data-connection-python.md)
17+
>
18+
19+
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. In this article, it gives you an End-to-End example about how to ingest data from a blob storage into Azure Data Explorer. You will learn how to programmatically create a resource group, azure resources (a storage account, an event hub, an Azure Data Explorer cluster), and how to configure Azure Data Explorer to ingest data from an storage account.
20+
21+
## Prerequisites
22+
23+
If you don't have an Azure subscription, create a [free Azure account](https://azure.microsoft.com/free/) before you begin.
24+
25+
## Install C# Nuget
26+
27+
* Install the [Microsoft.Azure.Management.kusto](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
28+
* Install the [Microsoft.Azure.Management.ResourceManager](https://www.nuget.org/packages/Microsoft.Azure.Management.ResourceManager).
29+
* Install the [Microsoft.Azure.Management.EventGrid](https://www.nuget.org/packages/Microsoft.Azure.Management.EventGrid/).
30+
* Install the [Microsoft.Azure.Storage.Blob](https://www.nuget.org/packages/Microsoft.Azure.Storage.Blob/).
31+
* Install the [Microsoft.Rest.ClientRuntime.Azure.Authentication](https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure.Authentication) for authentication.
32+
33+
[!INCLUDE [data-explorer-authentication](../../includes/data-explorer-authentication.md)]
34+
35+
[!INCLUDE [data-explorer-e2e-event-grid-resource-template](../../includes/data-explorer-e2e-event-grid-resource-template.md)]
36+
37+
## Code Example
38+
The following code example shows how to prepare all the relevant resources, and configurations for ingesting blobs into Azure Data Explorer step by step.
39+
40+
```csharp
41+
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
42+
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
43+
var clientSecret = "xxxxxxxxxxxxxx";//Client Secret
44+
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
45+
string location = "West Europe";
46+
string locationSmallCase = "westeurope";
47+
48+
49+
string deploymentName = "e2eexa2";
50+
string resourceGroupName = deploymentName + "resourcegroup";
51+
string eventHubName = deploymentName + "eventhub";
52+
string eventHubNamespaceName = eventHubName + "ns";
53+
string storageAccountName = deploymentName + "storage";
54+
string storageContainerName = deploymentName + "storagecontainer";
55+
string eventGridSubscriptionName = deploymentName + "eventgrid";
56+
string kustoClusterName = deploymentName + "kustocluster";
57+
string kustoDatabaseName = deploymentName + "kustodatabase";
58+
string kustoTableName = "Events";
59+
string kustoColumnMappingName = "Events_CSV_Mapping";
60+
string kustoDataConnectionName = deploymentName + "kustoeventgridconnection";
61+
62+
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
63+
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
64+
Console.WriteLine("Step 1: create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.");
65+
resourceManagementClient.SubscriptionId = subscriptionId;
66+
await resourceManagementClient.ResourceGroups.CreateOrUpdateAsync(resourceGroupName,
67+
new ResourceGroup() { Location = locationSmallCase });
68+
69+
Console.WriteLine(
70+
"Step 2: create a blob storage, a container in the storage account, an event hub, an azure data explorer cluster, and database by using an Azure Resource Manager template.");
71+
var parameters = $"{{\"eventHubNamespaceName\":{{\"value\":\"{eventHubNamespaceName}\"}},\"eventHubName\":{{\"value\":\"{eventHubName}\"}},\"storageAccountName\":{{\"value\":\"{storageAccountName}\"}},\"containerName\":{{\"value\":\"{storageContainerName}\"}},\"kustoClusterName\":{{\"value\":\"{kustoClusterName}\"}},\"kustoDatabaseName\":{{\"value\":\"{kustoDatabaseName}\"}}}}";
72+
string template = File.ReadAllText(@"C:\Users\lugoldbe\PycharmProjects\Test\template.json", Encoding.UTF8);
73+
await resourceManagementClient.Deployments.CreateOrUpdateAsync(resourceGroupName, deploymentName,
74+
new Deployment(new DeploymentProperties(DeploymentMode.Incremental, template: template,
75+
parameters: parameters)));
76+
77+
Console.WriteLine(
78+
"Step 3: create an event grid subscription to publish events of blobs created in a specific container to an event hub.");
79+
var eventGridClient = new EventGridManagementClient(serviceCreds)
80+
{
81+
SubscriptionId = subscriptionId
82+
};
83+
string storageResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName}";
84+
string eventHubResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{eventHubNamespaceName}/eventhubs/{eventHubName}";
85+
await eventGridClient.EventSubscriptions.CreateOrUpdateAsync(storageResourceId, eventGridSubscriptionName,
86+
new EventSubscription()
87+
{
88+
Destination = new EventHubEventSubscriptionDestination(eventHubResourceId),
89+
Filter = new EventSubscriptionFilter()
90+
{
91+
SubjectBeginsWith = $"/blobServices/default/containers/{storageContainerName}",
92+
IncludedEventTypes = new List<string>(){"Microsoft.Storage.BlobCreated"}
93+
}
94+
});
95+
96+
Console.WriteLine("Step 4: create a table and column mapping in Azure Data Explorer database.");
97+
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
98+
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
99+
{
100+
InitialCatalog = kustoDatabaseName,
101+
FederatedSecurity = true,
102+
ApplicationClientId = clientId,
103+
ApplicationKey = clientSecret,
104+
Authority = tenantId
105+
};
106+
107+
using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnectionStringBuilder))
108+
{
109+
var command =
110+
CslCommandGenerator.GenerateTableCreateCommand(
111+
kustoTableName,
112+
new[]
113+
{
114+
Tuple.Create("EventTime", "System.DateTime"),
115+
Tuple.Create("EventId", "System.Int32"),
116+
Tuple.Create("EventSummary", "System.String"),
117+
});
118+
119+
kustoClient.ExecuteControlCommand(command);
120+
121+
command = CslCommandGenerator.GenerateTableCsvMappingCreateCommand(
122+
kustoTableName,
123+
kustoColumnMappingName,
124+
new[]
125+
{
126+
new CsvColumnMapping { ColumnName = "EventTime", CslDataType="dateTime", Ordinal = 0 },
127+
new CsvColumnMapping { ColumnName = "EventId", CslDataType="int", Ordinal = 1 },
128+
new CsvColumnMapping { ColumnName = "EventSummary", CslDataType="string", Ordinal = 2 },
129+
});
130+
kustoClient.ExecuteControlCommand(command);
131+
}
132+
133+
Console.WriteLine("Step 5: add data connection.");
134+
var kustoManagementClient = new KustoManagementClient(serviceCreds)
135+
{
136+
SubscriptionId = subscriptionId
137+
};
138+
await kustoManagementClient.DataConnections.CreateOrUpdateAsync(resourceGroupName, kustoClusterName,
139+
kustoDatabaseName, dataConnectionName: kustoDataConnectionName, new EventGridDataConnection(storageResourceId, eventHubResourceId, consumerGroup: "$Default", location: location, tableName:kustoTableName, mappingRuleName: kustoColumnMappingName, dataFormat: "csv"));
140+
141+
```
142+
|**Setting** | **Suggested value** | **Field description**|
143+
|---|---|---|
144+
| tenantId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | Your tenant ID. Also known as directory ID.|
145+
| subscriptionId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
146+
| clientId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The client ID of the application that can access resources in your tenant.|
147+
| clientSecret | *xxxxxxxxxxxxxx* | The client secret of the application that can access resources in your tenant. |
148+
149+
## How to test the example
150+
1. Upload a file into the storage account
151+
152+
```csharp
153+
string storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxxxxxx;AccountKey=xxxxxxxxxxxxxx;EndpointSuffix=core.windows.net";
154+
var cloudStorageAccount = CloudStorageAccount.Parse(storageConnectionString);
155+
CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient();
156+
CloudBlobContainer container = blobClient.GetContainerReference(storageContainerName);
157+
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test.csv");
158+
var blobContent = @"2007-01-01 00:00:00.0000000,2592,Several trees down
159+
2007-01-01 00:00:00.0000000,4171,Winter Storm";
160+
await blockBlob.UploadTextAsync(blobContent);
161+
```
162+
|**Setting** | **Suggested value** | **Field description**|
163+
|---|---|---|
164+
| storageConnectionString | *xxxxxxxxxxxxxx* | The connection string of the programmatically created storage account.|
165+
166+
2. Run a test query in Azure Data Explorer
167+
```csharp
168+
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
169+
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
170+
{
171+
InitialCatalog = kustoDatabaseName,
172+
FederatedSecurity = true,
173+
ApplicationClientId = clientId,
174+
ApplicationKey = clientSecret,
175+
Authority = tenantId
176+
};
177+
using (var kustoClient = KustoClientFactory.CreateCslQueryProvider(kustoConnectionStringBuilder))
178+
{
179+
var query = $"{kustoTableName} | take 10";
180+
using (var reader = kustoClient.ExecuteQuery(query) as DataTableReader2)
181+
{// Print the contents of each of the result sets.
182+
while (reader.Read())
183+
{
184+
Console.WriteLine($"{reader[0]}, {reader[1]}, {reader[2]}");
185+
}
186+
}
187+
}
188+
```
189+
190+
## Clean up resources
191+
To delete the resource group and clean up resources, use the following command:
192+
193+
```csharp
194+
await resourceManagementClient.ResourceGroups.DeleteAsync(resourceGroupName);
195+
```

0 commit comments

Comments
 (0)