You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/outputs/azure_kusto.md
+54-54Lines changed: 54 additions & 54 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,87 +4,87 @@ description: Send logs to Azure Data Explorer (Kusto)
4
4
5
5
# Azure Data Explorer (Kusto)
6
6
7
-
The Kusto output plugin allows to ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, via the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics.
7
+
The _Kusto_ output plugin lets you ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, using the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics.
8
8
9
-
## For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database
9
+
## Ingest into Azure Data Explorer: create a Kusto cluster and database
10
10
11
-
You can create an Azure Data Explorer cluster in one of the following ways:
11
+
Create an Azure Data Explorer cluster in one of the following ways:
12
12
13
13
-[Create a free-tier cluster](https://dataexplorer.azure.com/freecluster)
14
14
-[Create a fully featured cluster](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal)
15
15
16
-
## For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database
16
+
## Ingest into Microsoft Fabric real time analytics: Creating an Eventhouse cluster and KQL database
17
17
18
-
You can create an Eventhouse cluster and a KQL database follow the following steps:
18
+
Create an Eventhouse cluster and a KQL database using the following steps:
19
19
20
20
-[Create an Eventhouse cluster](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-eventhouse-cluster)
21
21
-[Create a KQL database](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-database)
22
22
23
-
## Creating an Azure Registered Application
23
+
## Create an Azure registered application
24
24
25
-
Fluent-Bit will use the application's credentials, to ingest data into your cluster.
25
+
FluentBit uses the application's credentials to ingest data into your cluster.
26
26
27
-
-[Register an Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
27
+
-[Register an application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
28
28
-[Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret)
29
29
-[Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants)
30
30
31
-
## Creating a Table
31
+
## Create a table
32
32
33
-
Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:
33
+
FluentBit ingests the event data into Kusto in a JSON format. By default, the table includes 3 properties:
34
34
35
35
-`log` - the actual event payload.
36
36
-`tag` - the event tag.
37
37
-`timestamp` - the event timestamp.
38
38
39
39
A table with the expected schema must exist in order for data to be ingested properly.
By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping). The plugin can be configured to use an ingestion mapping via the `ingestion_mapping_reference` configuration key.
| tenant_id|_Required if `managed_identity_client_id` is not set_ - The tenant/domain ID of the AAD registered application. ||
54
-
| client_id|_Required if `managed_identity_client_id` is not set_ - The client ID of the AAD registered application. ||
55
-
| client_secret|_Required if `managed_identity_client_id` is not set_ - The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). |
56
-
| managed_identity_client_id|_Required if `tenant_id`, `client_id`, and `client_secret` are not set_ - The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (GUID) for user-assigned managed identity. ||
57
-
| ingestion_endpoint|_Required_ - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net|
58
-
| database_name|_Required_ - The database name. ||
59
-
| table_name|_Required_ - The table name. ||
60
-
| ingestion_mapping_reference |_Optional_ - The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. ||
61
-
| log_key| Key name of the log content. |`log`|
62
-
| include_tag_key| If enabled, a tag is appended to output. The key name is used `tag_key` property. |`On`|
63
-
| tag_key| The key name of tag. If `include_tag_key` is false, This property is ignored. |`tag`|
64
-
| include_time_key| If enabled, a timestamp is appended to output. The key name is used `time_key` property. |`On`|
65
-
| time_key| The key name of time. If `include_time_key` is false, This property is ignored.|`timestamp`|
66
-
| ingestion_endpoint_connect_timeout| The connection timeout of various Kusto endpoints in seconds.|`60`|
67
-
| compression_enabled| If enabled, sends compressed HTTP payload (gzip) to Kusto.|`true`|
68
-
| ingestion_resources_refresh_interval| The ingestion resources refresh interval of Kusto endpoint in seconds.
69
-
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
70
-
| buffering_enabled|_Optional_ - Enable buffering into disk before ingesting into Azure Kusto. |`Off`|
71
-
| buffer_dir|_Optional_ - When buffering is `On`, specifies the location of directory where the buffered data will be stored. |`/tmp/fluent-bit/azure-kusto/`|
72
-
| upload_timeout|_Optional_ - When buffering is `On`, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit. |`30m`|
73
-
| upload_file_size|_Optional_ - When buffering is `On`, specifies the size of files to be uploaded in MBs. |`200MB`|
74
-
| azure_kusto_buffer_key|_Optional_ - When buffering is `On`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. |`key`|
75
-
| store_dir_limit_size |_Optional_ - When buffering is `On`, set the max size of the buffer directory. |`8GB`|
76
-
| buffer_file_delete_early |_Optional_ - When buffering is `On`, whether to delete the buffered file early after successful blob creation. |`Off`|
77
-
| unify_tag |_Optional_ - This creates a single buffer file when the buffering mode is `On`. |`On`|
78
-
| blob_uri_length |_Optional_ - Set the length of generated blob URI before ingesting to Kusto. |`64`|
79
-
| scheduler_max_retries |_Optional_ - When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. |`3`|
80
-
| delete_on_max_upload_error |_Optional_ - When buffering is `On`, whether to delete the buffer file on maximum upload errors. |`Off`|
81
-
| io_timeout |_Optional_ - Configure the HTTP IO timeout for uploads. |`60s`|
82
-
83
-
### Configuration File
84
-
85
-
Get started quickly with this configuration file:
86
-
87
-
```
45
+
## Optional: create an ingestion mapping
46
+
47
+
By default, Kusto inserts incoming ingestion data into a table by inferring the mapped table columns from the payload properties. Mapping can be customized by creating a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping). The plugin can be configured to use an ingestion mapping with the `ingestion_mapping_reference` configuration key.
48
+
49
+
## Configuration parameters
50
+
51
+
| Key | Description | Default |
52
+
| --- | ----------- | ------- |
53
+
|`tenant_id`|The tenant/domain ID of the Azure Active Directory (AAD) registered application. Required if `managed_identity_client_id` isn't set. |_none_|
54
+
|`client_id`|The client ID of the AAD registered application. Required if `managed_identity_client_id` isn't set. |_none_|
55
+
|`client_secret`|The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). Required if `managed_identity_client_id` isn't set. |_none_|
56
+
|`managed_identity_client_id`|The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (`GUID`) for user-assigned managed identity. Required if `tenant_id`, `client_id`, and `client_secret` aren't set. |_none_|
57
+
|`ingestion_endpoint`|The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net`|_none_|
58
+
|`database_name`|The database name. |_none_|
59
+
|`table_name`|The table name. |_none_|
60
+
|`ingestion_mapping_reference`|Optional. The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. |_none_|
61
+
|`log_key`| Key name of the log content. |`log`|
62
+
|`include_tag_key`| If enabled, a tag is appended to output. The key name is used is the `tag_key` property. |`On`|
63
+
|`tag_key`| The key name of tag. If `include_tag_key` is `false`, this property is ignored. |`tag`|
64
+
|`include_time_key`| If enabled, a timestamp is appended to output. The key name is used is the `time_key` property. |`On`|
65
+
|`time_key`| The key name of `time`. If `include_time_key` is `false`, this property is ignored. |`timestamp`|
66
+
|`ingestion_endpoint_connect_timeout`| The connection timeout of various Kusto endpoints in seconds. |`60`|
67
+
|`compression_enabled`| If enabled, sends compressed HTTP payload (gzip) to Kusto. |`true`|
68
+
|`ingestion_resources_refresh_interval`| The ingestion resources refresh interval of Kusto endpoint in seconds.|_none_|
69
+
|`workers`| The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. |`0`|
70
+
|`buffering_enabled`| Optional. Enable buffering into disk before ingesting into Azure Kusto. |`Off`|
71
+
|`buffer_dir`| Optional. When buffering is `On`, specifies the location of directory where the buffered data will be stored. |`/tmp/fluent-bit/azure-kusto/`|
72
+
|`upload_timeout`| Optional. When buffering is `On`, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes ago and haven't reached `upload_file_size` limit. |`30m`|
73
+
|`upload_file_size`| Optional. When buffering is `On`, specifies the size of files to be uploaded in MB. |`200MB`|
74
+
|`azure_kusto_buffer_key`| Optional. When buffering is `On`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. |`key`|
75
+
|`store_dir_limit_size`|Optional. When buffering is `On`, set the max size of the buffer directory. |`8GB`|
76
+
|`buffer_file_delete_early`|Optional. When buffering is `On`, whether to delete the buffered file early after successful blob creation. |`Off`|
77
+
|`unify_tag`|Optional. This creates a single buffer file when the buffering mode is `On`. |`On`|
78
+
|`blob_uri_length`|Optional. Set the length of generated blob URI before ingesting to Kusto. |`64`|
79
+
|`scheduler_max_retries`|Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. |`3`|
80
+
|`delete_on_max_upload_error`|Optional. When buffering is `On`, whether to delete the buffer file on maximum upload errors. |`Off`|
81
+
|`io_timeout`|Optional. Configure the HTTP IO timeout for uploads. |`60s`|
0 commit comments