Skip to content

Commit d781acf

Browse files
lugoldbelugoldbe
authored andcommitted
cr fixes
1 parent d475c49 commit d781acf

File tree

4 files changed

+55
-45
lines changed

4 files changed

+55
-45
lines changed

articles/data-explorer/end-to-end-csharp.md

Lines changed: 24 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
11
---
2-
title: 'Create an End to End example for Azure Data Explorer by using Python'
3-
description: In this article, you learn how to use Azure Data Explorer with an End to End example using Python.
2+
title: 'End-to-end Blob ingestion into Azure Data Explorer using C#'
3+
description: In this article, you learn how to ingest blobs into Azure Data Explorer with an End to End example using C#.
44
author: lucygoldbergmicrosoft
55
ms.author: lugoldbe
66
ms.reviewer: orspodek
77
ms.service: data-explorer
88
ms.topic: conceptual
9-
ms.date: 09/24/2019
9+
ms.date: 10/23/2019
1010
---
1111

12-
# An End-to-End example for ingesting blobs into Azure Data Explorer using Python
12+
# End-to-end Blob ingestion into Azure Data Explorer using C#
1313

1414
> [!div class="op_single_selector"]
1515
> * [C#](end-to-end-csharp.md)
1616
> * [Python](end-to-end-python.md)
1717
>
1818
19-
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. In this article, it gives you an End-to-End example about how to ingest data from a blob storage into Azure Data Explorer. You will learn how to programmatically create a resource group, azure resources (a storage account, an event hub, an Azure Data Explorer cluster), and how to configure Azure Data Explorer to ingest data from a storage account.
19+
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. This article gives you an end-to-end example about how to ingest data from Blob Storage into Azure Data Explorer. You will learn how to programmatically create a resource group, a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You will also learn how to programmatically configure Azure Data Explorer to ingest data from the new storage account.
2020

2121
## Prerequisites
2222

@@ -34,8 +34,9 @@ If you don't have an Azure subscription, create a [free Azure account](https://a
3434

3535
[!INCLUDE [data-explorer-e2e-event-grid-resource-template](../../includes/data-explorer-e2e-event-grid-resource-template.md)]
3636

37-
## Code Example
38-
The following code example shows how to create azure resources, and configurations for ingesting blobs into Azure Data Explorer step by step.
37+
## Code example
38+
39+
The following code example gives you a step-by-step process resulting in data ingestion into Azure Data Explorer. You first create a resource group, and Azure resources such as a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You then create an Event Grid subscription and table and column mapping in the Azure Data Explorer database. Finally, you create the data connection to configure Azure Data Explorer to ingest data from the new storage account.
3940

4041
```csharp
4142
var tenantId = "xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx";//Directory (tenant) ID
@@ -61,21 +62,21 @@ string kustoDataConnectionName = deploymentName + "kustoeventgridconnection";
6162

6263
var serviceCreds = await ApplicationTokenProvider.LoginSilentAsync(tenantId, clientId, clientSecret);
6364
var resourceManagementClient = new ResourceManagementClient(serviceCreds);
64-
Console.WriteLine("Step 1: create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.");
65+
Console.WriteLine("Step 1: Create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.");
6566
resourceManagementClient.SubscriptionId = subscriptionId;
6667
await resourceManagementClient.ResourceGroups.CreateOrUpdateAsync(resourceGroupName,
6768
new ResourceGroup() { Location = locationSmallCase });
6869

6970
Console.WriteLine(
70-
"Step 2: create a blob storage, a container in the storage account, an event hub, an azure data explorer cluster, and database by using an Azure Resource Manager template.");
71+
"Step 2: Create a Blob Storage, a container in the Storage account, an Event Hub, an Azure Data Explorer cluster, and database by using an Azure Resource Manager template.");
7172
var parameters = $"{{\"eventHubNamespaceName\":{{\"value\":\"{eventHubNamespaceName}\"}},\"eventHubName\":{{\"value\":\"{eventHubName}\"}},\"storageAccountName\":{{\"value\":\"{storageAccountName}\"}},\"containerName\":{{\"value\":\"{storageContainerName}\"}},\"kustoClusterName\":{{\"value\":\"{kustoClusterName}\"}},\"kustoDatabaseName\":{{\"value\":\"{kustoDatabaseName}\"}}}}";
7273
string template = File.ReadAllText(azureResourceTemplatePath, Encoding.UTF8);
7374
await resourceManagementClient.Deployments.CreateOrUpdateAsync(resourceGroupName, deploymentName,
7475
new Deployment(new DeploymentProperties(DeploymentMode.Incremental, template: template,
7576
parameters: parameters)));
7677

7778
Console.WriteLine(
78-
"Step 3: create an event grid subscription to publish events of blobs created in a specific container to an event hub.");
79+
"Step 3: Create an Event Grid subscription to publish blob events created in a specific container to an Event Hub.");
7980
var eventGridClient = new EventGridManagementClient(serviceCreds)
8081
{
8182
SubscriptionId = subscriptionId
@@ -93,7 +94,7 @@ await eventGridClient.EventSubscriptions.CreateOrUpdateAsync(storageResourceId,
9394
}
9495
});
9596

96-
Console.WriteLine("Step 4: create a table (with three columns, EventTime, EventId, and EventSummary) and column mapping in Azure Data Explorer database.");
97+
Console.WriteLine("Step 4: Create a table (with three columns: EventTime, EventId, and EventSummary) and column mapping in your Azure Data Explorer database.");
9798
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
9899
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
99100
{
@@ -130,7 +131,7 @@ using (var kustoClient = KustoClientFactory.CreateCslAdminProvider(kustoConnecti
130131
kustoClient.ExecuteControlCommand(command);
131132
}
132133

133-
Console.WriteLine("Step 5: add a event grid data connection. Azure Data Explorer will automatically ingest the data when new blobs are created.");
134+
Console.WriteLine("Step 5: Add an Event Grid data connection. Azure Data Explorer will automatically ingest the data when new blobs are created.");
134135
var kustoManagementClient = new KustoManagementClient(serviceCreds)
135136
{
136137
SubscriptionId = subscriptionId
@@ -139,14 +140,15 @@ await kustoManagementClient.DataConnections.CreateOrUpdateAsync(resourceGroupNam
139140
kustoDatabaseName, dataConnectionName: kustoDataConnectionName, new EventGridDataConnection(storageResourceId, eventHubResourceId, consumerGroup: "$Default", location: location, tableName:kustoTableName, mappingRuleName: kustoColumnMappingName, dataFormat: "csv"));
140141

141142
```
142-
|**Setting** | **Suggested value** | **Field description**|
143+
| **Setting** | **Field description** |
143144
|---|---|---|
144-
| tenantId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | Your tenant ID. Also known as directory ID.|
145-
| subscriptionId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
146-
| clientId | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The client ID of the application that can access resources in your tenant.|
147-
| clientSecret | *xxxxxxxxxxxxxx* | The client secret of the application that can access resources in your tenant. |
145+
| tenantId | Your tenant ID. Also known as directory ID.|
146+
| subscriptionId | The subscription ID that you use for resource creation.|
147+
| clientId | The client ID of the application that can access resources in your tenant.|
148+
| clientSecret | The client secret of the application that can access resources in your tenant. |
149+
150+
## Test the code example
148151

149-
## How to test the example
150152
1. Upload a file into the storage account
151153

152154
```csharp
@@ -159,11 +161,12 @@ var blobContent = @"2007-01-01 00:00:00.0000000,2592,Several trees down
159161
2007-01-01 00:00:00.0000000,4171,Winter Storm";
160162
await blockBlob.UploadTextAsync(blobContent);
161163
```
162-
|**Setting** | **Suggested value** | **Field description**|
164+
|**Setting** | **Field description**|
163165
|---|---|---|
164-
| storageConnectionString | *xxxxxxxxxxxxxx* | The connection string of the programmatically created storage account.|
166+
| storageConnectionString | The connection string of the programmatically created storage account.|
165167

166168
2. Run a test query in Azure Data Explorer
169+
167170
```csharp
168171
var kustoUri = $"https://{kustoClusterName}.{locationSmallCase}.kusto.windows.net";
169172
var kustoConnectionStringBuilder = new KustoConnectionStringBuilder(kustoUri)
@@ -188,6 +191,7 @@ using (var kustoClient = KustoClientFactory.CreateCslQueryProvider(kustoConnecti
188191
```
189192

190193
## Clean up resources
194+
191195
To delete the resource group and clean up resources, use the following command:
192196

193197
```csharp

articles/data-explorer/end-to-end-python.md

Lines changed: 24 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,22 @@
11
---
2-
title: 'Create an End to End example for Azure Data Explorer by using Python'
3-
description: In this article, you learn how to use Azure Data Explorer with an End to End example using Python.
2+
title: 'End-to-end Blob ingestion into Azure Data Explorer using Python'
3+
description: In this article, you learn how to use ingest blobs into Azure Data Explorer with an End to End example using Python.
44
author: lucygoldbergmicrosoft
55
ms.author: lugoldbe
66
ms.reviewer: orspodek
77
ms.service: data-explorer
88
ms.topic: conceptual
9-
ms.date: 09/24/2019
9+
ms.date: 10/23/2019
1010
---
1111

12-
# An End-to-End example for ingesting blobs into Azure Data Explorer using Python
12+
# End-to-end Blob ingestion into Azure Data Explorer using Python
1313

1414
> [!div class="op_single_selector"]
1515
> * [C#](end-to-end-csharp.md)
1616
> * [Python](end-to-end-python.md)
1717
>
1818
19-
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. In this article, it gives you an End-to-End example about how to ingest data from a blob storage into Azure Data Explorer. You will learn how to programmatically create a resource group, azure resources (a storage account, an event hub, an Azure Data Explorer cluster), and how to configure Azure Data Explorer to ingest data from a storage account.
19+
Azure Data Explorer is a fast and scalable data exploration service for log and telemetry data. This article gives you an end-to-end example about how to ingest data from Blob Storage into Azure Data Explorer. You will learn how to programmatically create a resource group, a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You will also learn how to programmatically configure Azure Data Explorer to ingest data from the new storage account.
2020

2121
## Prerequisites
2222

@@ -37,8 +37,9 @@ pip install azure-storage-blob
3737

3838
[!INCLUDE [data-explorer-e2e-event-grid-resource-template](../../includes/data-explorer-e2e-event-grid-resource-template.md)]
3939

40-
## Code Example
41-
The following code example shows how to create azure resources, and configurations for ingesting blobs into Azure Data Explorer step by step.
40+
## Code example
41+
42+
The following code example gives you a step-by-step process resulting in data ingestion into Azure Data Explorer. You first create a resource group, and Azure resources such as a storage account and container, an Event Hub, and an Azure Data Explorer cluster and database. You then create an Event Grid subscription and table and column mapping in the Azure Data Explorer database. Finally, you create the data connection to configure Azure Data Explorer to ingest data from the new storage account.
4243

4344
```python
4445
from azure.common.credentials import ServicePrincipalCredentials
@@ -84,15 +85,15 @@ credentials = ServicePrincipalCredentials(
8485
)
8586
resource_client = ResourceManagementClient(credentials, subscription_id)
8687

87-
print('Step 1: create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.')
88+
print('Step 1: Create a new resource group in your Azure subscription to manage all the resources for using Azure Data Explorer.')
8889
resource_client.resource_groups.create_or_update(
8990
resource_group_name,
9091
{
9192
'location': location_small_case
9293
}
9394
)
9495

95-
print('Step 2: create a blob storage, a container in the storage account, an event hub, an azure data explorer cluster, and database by using an Azure Resource Manager template.')
96+
print('Step 2: Create a Blob Storage, a container in the Storage account, an Event Hub, an Azure Data Explorer cluster, and database by using an Azure Resource Manager template.')
9697
#Read the Azure Resource Manager template
9798
with open(azure_resource_template_path, 'r') as template_file_fd:
9899
template = json.load(template_file_fd)
@@ -120,7 +121,7 @@ poller = resource_client.deployments.create_or_update(
120121
)
121122
poller.wait()
122123

123-
print('Step 3: create an event grid subscription to publish events of blobs created in a specific container to an event hub.')
124+
print('Step 3: Create an Event Grid subscription to publish blob events created in a specific container to an Event Hub.')
124125
event_client = EventGridManagementClient(credentials, subscription_id)
125126
storage_resource_id = '/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Storage/storageAccounts/{}'.format(subscription_id, resource_group_name, storage_account_name)
126127
event_hub_resource_id = '/subscriptions/{}/resourceGroups/{}/providers/Microsoft.EventHub/namespaces/{}/eventhubs/{}'.format(subscription_id, resource_group_name, event_hub_namespace_name, event_hub_name)
@@ -139,7 +140,7 @@ event_client.event_subscriptions.create_or_update(storage_resource_id, event_gri
139140
})
140141

141142

142-
print('Step 4: create a table (with three columns, EventTime, EventId, and EventSummary) and column mapping in Azure Data Explorer database.')
143+
print('Step 4: Create a table (with three columns: EventTime, EventId, and EventSummary) and column mapping in your Azure Data Explorer database.')
143144
kusto_uri = "https://{}.{}.kusto.windows.net".format(kusto_cluster_name, location_small_case)
144145
database_name = kusto_database_name
145146
kusto_connection_string_builder = KustoConnectionStringBuilder.with_aad_application_key_authentication(connection_string=kusto_uri, aad_app_id=client_id, app_key=client_secret, authority_id=tenant_id)
@@ -152,7 +153,7 @@ create_column_mapping_command = ".create table " + kusto_table_name + " ingestio
152153
kusto_client.execute_mgmt(database_name, create_column_mapping_command)
153154

154155

155-
print('Step 5: add a event grid data connection. Azure Data Explorer will automatically ingest the data when new blobs are created.')
156+
print('Step 5: Add an Event Grid data connection. Azure Data Explorer will automatically ingest the data when new blobs are created.')
156157
kusto_management_client = KustoManagementClient(credentials, subscription_id)
157158
data_connections = kusto_management_client.data_connections
158159
#Returns an instance of LROPoller, see https://docs.microsoft.com/python/api/msrest/msrest.polling.lropoller?view=azure-python
@@ -161,14 +162,15 @@ poller = data_connections.create_or_update(resource_group_name=resource_group_na
161162
event_hub_resource_id=event_hub_resource_id, consumer_group="$Default", location=location, table_name=kusto_table_name, mapping_rule_name=kusto_column_mapping_name, data_format="csv"))
162163
poller.wait()
163164
```
164-
|**Setting** | **Suggested value** | **Field description**|
165+
|**Setting** | **Field description**|
165166
|---|---|---|
166-
| tenant_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | Your tenant ID. Also known as directory ID.|
167-
| subscription_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The subscription ID that you use for resource creation.|
168-
| client_id | *xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx* | The client ID of the application that can access resources in your tenant.|
169-
| client_secret | *xxxxxxxxxxxxxx* | The client secret of the application that can access resources in your tenant. |
167+
| tenant_id | Your tenant ID. Also known as directory ID.|
168+
| subscription_id | The subscription ID that you use for resource creation.|
169+
| client_id | The client ID of the application that can access resources in your tenant.|
170+
| client_secret | The client secret of the application that can access resources in your tenant. |
171+
172+
## Test the code example
170173

171-
## How to test the example
172174
1. Upload a file into the storage account
173175

174176
```python
@@ -179,11 +181,12 @@ blob_content = """2007-01-01 00:00:00.0000000,2592,Several trees down
179181
2007-01-01 00:00:00.0000000,4171,Winter Storm"""
180182
block_blob_service.create_blob_from_text(container_name=storage_container_name, blob_name=blob_name, text=blob_content)
181183
```
182-
|**Setting** | **Suggested value** | **Field description**|
184+
|**Setting** | **Field description**|
183185
|---|---|---|
184-
| account_key | *xxxxxxxxxxxxxx* | The access key of the programmatically created storage account.|
186+
| account_key | The access key of the programmatically created storage account.|
185187

186188
2. Run a test query in Azure Data Explorer
189+
187190
```python
188191
kusto_uri = "https://{}.{}.kusto.windows.net".format(kusto_cluster_name, location_small_case)
189192
kusto_connection_string_builder = KustoConnectionStringBuilder.with_aad_application_key_authentication(connection_string=kusto_uri, aad_app_id=client_id, app_key=client_secret, authority_id=tenant_id)
@@ -194,6 +197,7 @@ print(response.primary_results[0].rows_count)
194197
```
195198

196199
## Clean up resources
200+
197201
To delete the resource group and clean up resources, use the following command:
198202

199203
```python

articles/data-explorer/toc.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,14 @@
1313
href: ingest-sample-data.md
1414
- name: Query data with Web UI
1515
href: web-query-data.md
16-
- name: An End to End Example
16+
- name: Tutorials
17+
items:
18+
- name: E2E Programmatic Blob ingestion
1719
items:
1820
- name: C#
1921
href: end-to-end-csharp.md
20-
- name: python
22+
- name: Python
2123
href: end-to-end-python.md
22-
- name: Tutorials
23-
items:
2424
- name: Ingest data without using code
2525
href: ingest-data-no-code.md
2626
- name: Visualize data in Power BI

includes/data-explorer-e2e-event-grid-resource-template.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,14 @@
22
author: lugoldbemicrosoft
33
ms.service: data-explorer
44
ms.topic: include
5-
ms.date: 10/10/2019
5+
ms.date: 10/23/2019
66
ms.author: lugoldbe
77
---
88

99
## Azure Resource Manager template
10+
1011
In this article, an Azure Resource Manager template is used to create all the azure resources, a storage account, a container, an event hub, an azure data explorer cluster, and a database. Save the following content into a file with name `template.json`, which will be used to run the code example.
12+
1113
```json
1214
{
1315
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

0 commit comments

Comments
 (0)