Skip to content

Commit 222e103

Browse files
ShannonLeavittShannonLeavitt
authored
Merge pull request #91823 from lucygoldbergmicrosoft/addkustoe2eexample
add an end to end example for kusto
2 parents 50a5a5b + d781acf commit 222e103

7 files changed

+596
-5
lines changed

‎articles/data-explorer/create-cluster-database-csharp.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@ Azure Data Explorer is a fast, fully managed data analytics service for real-tim
2929

3030
## Install C# Nuget
3131

32-
1. Install the [Azure Data Explorer (Kusto) nuget package](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
32+
* Install the [Azure Data Explorer (Kusto) nuget package](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
3333

34-
1. Install the [Microsoft.IdentityModel.Clients.ActiveDirectory nuget package](https://www.nuget.org/packages/Microsoft.IdentityModel.Clients.ActiveDirectory/) for authentication.
34+
* Install the [Microsoft.IdentityModel.Clients.ActiveDirectory nuget package](https://www.nuget.org/packages/Microsoft.IdentityModel.Clients.ActiveDirectory/) for authentication.
3535

3636
## Authentication
3737
For running the examples in this article, we need an Azure AD Application and service principal that can access resources. Check [create an Azure AD application](https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal) to create a free Azure AD Application and add role assignment at the subscription scope. It also shows how to get the `Directory (tenant) ID`, `Application ID`, and `Client Secret`.

‎articles/data-explorer/data-connection-event-grid-python.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ poller = kusto_management_client.data_connections.create_or_update(resource_grou
8181
|**Setting** | **Suggested value** | **Field description**|
8282
|---|---|---|
8383
| tenant_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | Your tenant ID. Also known as directory ID.|
84-
| subscriptionId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
84+
| subscription_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
8585
| client_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The client ID of the application that can access resources in your tenant.|
8686
| client_secret | *xxxxxxxxxxxxxx* | The client secret of the application that can access resources in your tenant. |
8787
| resource_group_name | *testrg* | The name of the resource group containing your cluster.|

‎articles/data-explorer/end-to-end-csharp.md

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
---
2+
title: 'End-to-end Blob ingestion into Azure Data Explorer using C#'
3+
description: In this article, you learn how to ingest blobs into Azure Data Explorer with an End to End example using C#.
4+
author: lucygoldbergmicrosoft
5+
ms.author: lugoldbe
6+
ms.reviewer: orspodek
7+
ms.service: data-explorer
8+
ms.topic: conceptual
9+
ms.date: 10/23/2019
10+
---
11+
12+
# End-to-end Blob ingestion into Azure Data Explorer using C#
13+
14+
> [!div class="op_single_selector"]
15+
> * [C#](end-to-end-csharp.md)
16+
> * [Python](end-to-end-python.md)
17+
>
18+
19+
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. This article gives you an end-to-end example about how to ingest data from Blob Storage into Azure Data Explorer. You will learn how to programmatically create a resource group, a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You will also learn how to programmatically configure Azure Data Explorer to ingest data from the new storage account.
20+
21+
## Prerequisites
22+
23+
If you don't have an Azure subscription, create a [free Azure account](https://azure.microsoft.com/free/) before you begin.
24+
25+
## Install C# Nuget
26+
27+
* Install the [Microsoft.Azure.Management.kusto](https://www.nuget.org/packages/Microsoft.Azure.Management.Kusto/).
28+
* Install the [Microsoft.Azure.Management.ResourceManager](https://www.nuget.org/packages/Microsoft.Azure.Management.ResourceManager).
29+
* Install the [Microsoft.Azure.Management.EventGrid](https://www.nuget.org/packages/Microsoft.Azure.Management.EventGrid/).
30+
* Install the [Microsoft.Azure.Storage.Blob](https://www.nuget.org/packages/Microsoft.Azure.Storage.Blob/).
31+
* Install the [Microsoft.Rest.ClientRuntime.Azure.Authentication](https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure.Authentication) for authentication.
32+
33+
[!INCLUDE [data-explorer-authentication](../../includes/data-explorer-authentication.md)]
34+
35+
[!INCLUDE [data-explorer-e2e-event-grid-resource-template](../../includes/data-explorer-e2e-event-grid-resource-template.md)]
36+
37+
## Code example
38+
39+
The following code example gives you a step-by-step process resulting in data ingestion into Azure Data Explorer. You first create a resource group, and Azure resources such as a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You then create an Event Grid subscription and table and column mapping in the Azure Data Explorer database. Finally, you create the data connection to configure Azure Data Explorer to ingest data from the new storage account.
40+
41+
```csharp
42+
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
43+
var clientId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Application ID
44+
var clientSecret = "xxxxxxxxxxxxxx";//Client Secret
45+
var subscriptionId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";
46+
string location = "West Europe";
47+
string locationSmallCase = "westeurope";
48+
string azureResourceTemplatePath = @"xxxxxxxxx\template.json";//path to the Azure Resource Manager template json from the previous section
49+
50+
string deploymentName = "e2eexample";
51+
string resourceGroupName = deploymentName + "resourcegroup";
52+
string eventHubName = deploymentName + "eventhub";
53+
string eventHubNamespaceName = eventHubName + "ns";
54+
string storageAccountName = deploymentName + "storage";
55+
string storageContainerName = deploymentName + "storagecontainer";
56+
string eventGridSubscriptionName = deploymentName + "eventgrid";
57+
string kustoClusterName = deploymentName + "kustocluster";
58+
string kustoDatabaseName = deploymentName + "kustodatabase";
59+
string kustoTableName = "Events";
60+
string kustoColumnMappingName = "Events_CSV_Mapping";
61+
string kustoDataConnectionName = deploymentName + "kustoeventgridconnection";
62+
63+
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
64+
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
65+
Console.WriteLine("Step 1: Create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.");
66+
resourceManagementClient.SubscriptionId = subscriptionId;
67+
await resourceManagementClient.ResourceGroups.CreateOrUpdateAsync(resourceGroupName,
68+
new ResourceGroup() { Location = locationSmallCase });
69+
70+
Console.WriteLine(
71+
"Step 2: Create a Blob Storage, a container in the Storage account, an Event Hub, an Azure Data Explorer cluster, and database by using an Azure Resource Manager template.");
72+
var parameters = $"{{\"eventHubNamespaceName\":{{\"value\":\"{eventHubNamespaceName}\"}},\"eventHubName\":{{\"value\":\"{eventHubName}\"}},\"storageAccountName\":{{\"value\":\"{storageAccountName}\"}},\"containerName\":{{\"value\":\"{storageContainerName}\"}},\"kustoClusterName\":{{\"value\":\"{kustoClusterName}\"}},\"kustoDatabaseName\":{{\"value\":\"{kustoDatabaseName}\"}}}}";
73+
string template = File.ReadAllText(azureResourceTemplatePath, Encoding.UTF8);
74+
await resourceManagementClient.Deployments.CreateOrUpdateAsync(resourceGroupName, deploymentName,
75+
new Deployment(new DeploymentProperties(DeploymentMode.Incremental, template: template,
76+
parameters: parameters)));
77+
78+
Console.WriteLine(
79+
"Step 3: Create an Event Grid subscription to publish blob events created in a specific container to an Event Hub.");
80+
var eventGridClient = new EventGridManagementClient(serviceCreds)
81+
{
82+
SubscriptionId = subscriptionId
83+
};
84+
string storageResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName}";
85+
string eventHubResourceId = $"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{eventHubNamespaceName}/eventhubs/{eventHubName}";
86+
await eventGridClient.EventSubscriptions.CreateOrUpdateAsync(storageResourceId, eventGridSubscriptionName,
87+
new EventSubscription()
88+
{
89+
Destination = new EventHubEventSubscriptionDestination(eventHubResourceId),
90+
Filter = new EventSubscriptionFilter()
91+
{
92+
SubjectBeginsWith = $"/blobServices/default/containers/{storageContainerName}",
93+
IncludedEventTypes = new List<string>(){"Microsoft.Storage.BlobCreated"}
94+
}
95+
});
96+
97+
Console.WriteLine("Step 4: Create a table (with three columns: EventTime, EventId, and EventSummary) and column mapping in your Azure Data Explorer database.");
98+
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
99+
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
100+
{
101+
InitialCatalog = kustoDatabaseName,
102+
FederatedSecurity = true,
103+
ApplicationClientId = clientId,
104+
ApplicationKey = clientSecret,
105+
Authority = tenantId
106+
};
107+
108+
using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnectionStringBuilder))
109+
{
110+
var command =
111+
CslCommandGenerator.GenerateTableCreateCommand(
112+
kustoTableName,
113+
new[]
114+
{
115+
Tuple.Create("EventTime", "System.DateTime"),
116+
Tuple.Create("EventId", "System.Int32"),
117+
Tuple.Create("EventSummary", "System.String"),
118+
});
119+
120+
kustoClient.ExecuteControlCommand(command);
121+
122+
command = CslCommandGenerator.GenerateTableCsvMappingCreateCommand(
123+
kustoTableName,
124+
kustoColumnMappingName,
125+
new[]
126+
{
127+
new CsvColumnMapping { ColumnName = "EventTime", CslDataType="dateTime", Ordinal = 0 },
128+
new CsvColumnMapping { ColumnName = "EventId", CslDataType="int", Ordinal = 1 },
129+
new CsvColumnMapping { ColumnName = "EventSummary", CslDataType="string", Ordinal = 2 },
130+
});
131+
kustoClient.ExecuteControlCommand(command);
132+
}
133+
134+
Console.WriteLine("Step 5: Add an Event Grid data connection. Azure Data Explorer will automatically ingest the data when new blobs are created.");
135+
var kustoManagementClient = new KustoManagementClient(serviceCreds)
136+
{
137+
SubscriptionId = subscriptionId
138+
};
139+
await kustoManagementClient.DataConnections.CreateOrUpdateAsync(resourceGroupName, kustoClusterName,
140+
kustoDatabaseName, dataConnectionName: kustoDataConnectionName, new EventGridDataConnection(storageResourceId, eventHubResourceId, consumerGroup: "$Default", location: location, tableName:kustoTableName, mappingRuleName: kustoColumnMappingName, dataFormat: "csv"));
141+
142+
```
143+
| **Setting** | **Field description** |
144+
|---|---|---|
145+
| tenantId | Your tenant ID. Also known as directory ID.|
146+
| subscriptionId | The subscription ID that you use for resource creation.|
147+
| clientId | The client ID of the application that can access resources in your tenant.|
148+
| clientSecret | The client secret of the application that can access resources in your tenant. |
149+
150+
## Test the code example
151+
152+
1. Upload a file into the storage account
153+
154+
```csharp
155+
string storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxxxxxxx;AccountKey=xxxxxxxxxxxxxx;EndpointSuffix=core.windows.net";
156+
var cloudStorageAccount = CloudStorageAccount.Parse(storageConnectionString);
157+
CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient();
158+
CloudBlobContainer container = blobClient.GetContainerReference(storageContainerName);
159+
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test.csv");
160+
var blobContent = @"2007-01-01 00:00:00.0000000,2592,Several trees down
161+
2007-01-01 00:00:00.0000000,4171,Winter Storm";
162+
await blockBlob.UploadTextAsync(blobContent);
163+
```
164+
|**Setting** | **Field description**|
165+
|---|---|---|
166+
| storageConnectionString | The connection string of the programmatically created storage account.|
167+
168+
2. Run a test query in Azure Data Explorer
169+
170+
```csharp
171+
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
172+
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
173+
{
174+
InitialCatalog = kustoDatabaseName,
175+
FederatedSecurity = true,
176+
ApplicationClientId = clientId,
177+
ApplicationKey = clientSecret,
178+
Authority = tenantId
179+
};
180+
using (var kustoClient = KustoClientFactory.CreateCslQueryProvider(kustoConnectionStringBuilder))
181+
{
182+
var query = $"{kustoTableName} | take 10";
183+
using (var reader = kustoClient.ExecuteQuery(query) as DataTableReader2)
184+
{// Print the contents of each of the result sets.
185+
while (reader.Read())
186+
{
187+
Console.WriteLine($"{reader[0]}, {reader[1]}, {reader[2]}");
188+
}
189+
}
190+
}
191+
```
192+
193+
## Clean up resources
194+
195+
To delete the resource group and clean up resources, use the following command:
196+
197+
```csharp
198+
await resourceManagementClient.ResourceGroups.DeleteAsync(resourceGroupName);
199+
```
200+
201+
## Next steps
202+
203+
* [Create an Azure Data Explorer cluster and database](create-cluster-database-csharp.md) to learn about other ways to create a cluster and database.
204+
* [Azure Data Explorer data ingestion](ingest-data-overview.md) to learn more about ingestion methods.
205+
* [Quickstart: Query data in Azure Data Explorer](web-query-data.md) Web UI.
206+
* [Write queries](write-queries.md) with Kusto Query Language.

0 commit comments

Comments
 (0)