diff --git a/pipeline/outputs/azure_kusto.md b/pipeline/outputs/azure_kusto.md index 361e4a2cc..f9483b09a 100644 --- a/pipeline/outputs/azure_kusto.md +++ b/pipeline/outputs/azure_kusto.md @@ -4,33 +4,33 @@ description: Send logs to Azure Data Explorer (Kusto) # Azure Data Explorer (Kusto) -The Kusto output plugin allows to ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, via the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics. +The _Kusto_ output plugin lets you ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, using the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics. -## For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database +## Ingest into Azure Data Explorer: create a Kusto cluster and database -You can create an Azure Data Explorer cluster in one of the following ways: +Create an Azure Data Explorer cluster in one of the following ways: - [Create a free-tier cluster](https://dataexplorer.azure.com/freecluster) - [Create a fully featured cluster](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) -## For ingesting into Microsoft Fabric Real Time Analytics : Creating an Eventhouse Cluster and KQL Database +## Ingest into Microsoft Fabric real time analytics: Creating an Eventhouse cluster and KQL database -You can create an Eventhouse cluster and a KQL database follow the following steps: +Create an Eventhouse cluster and a KQL database using the following steps: - [Create an Eventhouse cluster](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-eventhouse-cluster) - [Create a KQL database](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-database) -## Creating an Azure Registered Application +## Create an Azure registered application -Fluent-Bit will use the application's credentials, to ingest data into your cluster. +Fluent Bit uses the application's credentials to ingest data into your cluster. -- [Register an Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application) +- [Register an application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application) - [Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret) - [Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants) -## Creating a Table +## Create a table -Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties: +Fluent Bit ingests the event data into Kusto in a JSON format. By default, the table includes 3 properties: - `log` - the actual event payload. - `tag` - the event tag. @@ -38,53 +38,53 @@ Fluent-Bit ingests the event data into Kusto in a JSON format, that by default w A table with the expected schema must exist in order for data to be ingested properly. -```kql +```text .create table FluentBit (log:dynamic, tag:string, timestamp:datetime) ``` -## Optional - Creating an Ingestion Mapping - -By default, Kusto will insert incoming ingestions into a table by inferring the mapped table columns, from the payload properties. However, this mapping can be customized by creatng a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping). The plugin can be configured to use an ingestion mapping via the `ingestion_mapping_reference` configuration key. - -## Configuration Parameters - -| Key | Description | Default | -| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | -| tenant_id | _Required if `managed_identity_client_id` is not set_ - The tenant/domain ID of the AAD registered application. | | -| client_id | _Required if `managed_identity_client_id` is not set_ - The client ID of the AAD registered application. | | -| client_secret | _Required if `managed_identity_client_id` is not set_ - The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). | -| managed_identity_client_id | _Required if `tenant_id`, `client_id`, and `client_secret` are not set_ - The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (GUID) for user-assigned managed identity. | | -| ingestion_endpoint | _Required_ - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net | -| database_name | _Required_ - The database name. | | -| table_name | _Required_ - The table name. | | -| ingestion_mapping_reference | _Optional_ - The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | | -| log_key | Key name of the log content. | `log` | -| include_tag_key | If enabled, a tag is appended to output. The key name is used `tag_key` property. | `On` | -| tag_key | The key name of tag. If `include_tag_key` is false, This property is ignored. | `tag` | -| include_time_key | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `On` | -| time_key | The key name of time. If `include_time_key` is false, This property is ignored. | `timestamp` | -| ingestion_endpoint_connect_timeout | The connection timeout of various Kusto endpoints in seconds. | `60` | -| compression_enabled | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | -| ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. -| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | -| buffering_enabled | _Optional_ - Enable buffering into disk before ingesting into Azure Kusto. | `Off` | -| buffer_dir | _Optional_ - When buffering is `On`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | -| upload_timeout | _Optional_ - When buffering is `On`, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached `upload_file_size` limit. | `30m` | -| upload_file_size | _Optional_ - When buffering is `On`, specifies the size of files to be uploaded in MBs. | `200MB` | -| azure_kusto_buffer_key | _Optional_ - When buffering is `On`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. | `key` | -| store_dir_limit_size | _Optional_ - When buffering is `On`, set the max size of the buffer directory. | `8GB` | -| buffer_file_delete_early | _Optional_ - When buffering is `On`, whether to delete the buffered file early after successful blob creation. | `Off` | -| unify_tag | _Optional_ - This creates a single buffer file when the buffering mode is `On`. | `On` | -| blob_uri_length | _Optional_ - Set the length of generated blob URI before ingesting to Kusto. | `64` | -| scheduler_max_retries | _Optional_ - When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | -| delete_on_max_upload_error | _Optional_ - When buffering is `On`, whether to delete the buffer file on maximum upload errors. | `Off` | -| io_timeout | _Optional_ - Configure the HTTP IO timeout for uploads. | `60s` | - -### Configuration File - -Get started quickly with this configuration file: - -``` +## Optional: create an ingestion mapping + +By default, Kusto inserts incoming ingestion data into a table by inferring the mapped table columns from the payload properties. Mapping can be customized by creating a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping). The plugin can be configured to use an ingestion mapping with the `ingestion_mapping_reference` configuration key. + +## Configuration parameters + +| Key | Description | Default | +| --- | ----------- | ------- | +| `tenant_id` | The tenant/domain ID of the Azure Active Directory (AAD) registered application. Required if `managed_identity_client_id` isn't set. | _none_ | +| `client_id` | The client ID of the AAD registered application. Required if `managed_identity_client_id` isn't set. | _none_ | +| `client_secret` | The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). Required if `managed_identity_client_id` isn't set. | _none_ | +| `managed_identity_client_id` | The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (`GUID`) for user-assigned managed identity. Required if `tenant_id`, `client_id`, and `client_secret` aren't set. | _none_ | +| `ingestion_endpoint` | The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net` | _none_ | +| `database_name` | The database name. | _none_ | +| `table_name` | The table name. | _none_ | +| `ingestion_mapping_reference` | Optional. The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | _none_ | +| `log_key` | Key name of the log content. | `log` | +| `include_tag_key` | If enabled, a tag is appended to output. The key name is used is the `tag_key` property. | `On` | +| `tag_key` | The key name of tag. If `include_tag_key` is `false`, this property is ignored. | `tag` | +| `include_time_key` | If enabled, a timestamp is appended to output. The key name is used is the `time_key` property. | `On` | +| `time_key` | The key name of `time`. If `include_time_key` is `false`, this property is ignored. | `timestamp` | +| `ingestion_endpoint_connect_timeout` | The connection timeout of various Kusto endpoints in seconds. | `60` | +| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` | +| `ingestion_resources_refresh_interval` | The ingestion resources refresh interval of Kusto endpoint in seconds. | _none_ | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | +| `buffering_enabled` | Optional. Enable buffering into disk before ingesting into Azure Kusto. | `Off` | +| `buffer_dir` | Optional. When buffering is `On`, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` | +| `upload_timeout` | Optional. When buffering is `On`, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes ago and haven't reached `upload_file_size` limit. | `30m` | +| `upload_file_size` | Optional. When buffering is `On`, specifies the size of files to be uploaded in MB. | `200MB` | +| `azure_kusto_buffer_key` | Optional. When buffering is `On`, set the Azure Kusto buffer key which must be specified when using multiple instances of Azure Kusto output plugin and buffering is enabled. | `key` | +| `store_dir_limit_size` | Optional. When buffering is `On`, set the max size of the buffer directory. | `8GB` | +| `buffer_file_delete_early` | Optional. When buffering is `On`, whether to delete the buffered file early after successful blob creation. | `Off` | +| `unify_tag` | Optional. This creates a single buffer file when the buffering mode is `On`. | `On` | +| `blob_uri_length` | Optional. Set the length of generated blob URI before ingesting to Kusto. | `64` | +| `scheduler_max_retries` | Optional. When buffering is `On`, set the maximum number of retries for ingestion using the scheduler. | `3` | +| `delete_on_max_upload_error` | Optional. When buffering is `On`, whether to delete the buffer file on maximum upload errors. | `Off` | +| `io_timeout` | Optional. Configure the HTTP IO timeout for uploads. | `60s` | + +### Configuration file + +Get started with this configuration file: + +```text [OUTPUT] Match * Name azure_kusto diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml index 954c546a8..8273f714b 100644 --- a/vale-styles/FluentBit/Acronyms.yml +++ b/vale-styles/FluentBit/Acronyms.yml @@ -43,14 +43,15 @@ exceptions: - GZIP - HEC - HPA - - IAM - HTML - HTTP - HTTPS + - IAM - IDE - JAR - JSON - JSX + - KQL - LESS - LLDB - LLVM diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 54216e128..d22ed1fce 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -64,6 +64,7 @@ endcode endhint endtab endtabs +Eventhouse Exabeam Fargate Firehose