Skip to content

Commit e2f15be

Browse files
Update azure_kusto.md for buffering commits
Signed-off-by: Tanmaya Panda <[email protected]>
1 parent e4f6a3e commit e2f15be

File tree

1 file changed

+26
-0
lines changed

1 file changed

+26
-0
lines changed

pipeline/outputs/azure_kusto.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,19 @@ By default, Kusto will insert incoming ingestions into a table by inferring the
6464
| include_time_key | If enabled, a timestamp is appended to output. The key name is used `time_key` property. | `On` |
6565
| time_key | The key name of time. If `include_time_key` is false, This property is ignored. | `timestamp` |
6666
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
67+
| buffering_enabled | _Optional_ - Enable buffering into disk before ingesting into Azure Kusto. | `Off` |
68+
| buffer_dir | _Optional_ - When buffering is turned ON, specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-kusto/` |
69+
| upload_timeout | _Optional_ - When buffering is turned ON, specifies a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than x minutes and haven't reached upload_file_size limit yet. | `30m` |
70+
| upload_file_size | _Optional_ - When buffering is turned ON, specifies the size of files to be uploaded in MBs. | `200MB` |
71+
| azure_kusto_buffer_key | _Optional_ - When buffering is turned ON, set the azure kusto buffer key which needs to be specified when using multiple instances of azure kusto output plugin and buffering is enabled. | `key` |
72+
| store_dir_limit_size | _Optional_ - When buffering is turned ON, set the max size of the buffer directory. | `8GB` |
73+
| buffer_file_delete_early | _Optional_ - When buffering is turned ON, whether to delete the buffered file early after successful blob creation. | `Off` |
74+
| unify_tag | _Optional_ - This creates a single buffer file when the buffering mode is ON. | `On` |
75+
| blob_uri_length | _Optional_ - Set the length of generated blob uri before ingesting to kusto. | `64` |
76+
| scheduler_max_retries | _Optional_ - When buffering is turned ON, Set the maximum number of retries for ingestion using the scheduler. | `3` |
77+
| use_imds | _Optional_ - Whether to use IMDS to retrieve oauth token. | `Off` |
78+
| delete_on_max_upload_error | _Optional_ - When buffering is turned ON, Whether to delete the buffer file on maximum upload errors. | `Off` |
79+
| io_timeout | _Optional_ - Configure the HTTP IO timeout for uploads. | `60s` |
6780

6881
### Configuration File
6982

@@ -80,6 +93,19 @@ Get started quickly with this configuration file:
8093
Database_Name <database_name>
8194
Table_Name <table_name>
8295
Ingestion_Mapping_Reference <mapping_name>
96+
buffering_enabled On
97+
upload_timeout 2m
98+
upload_file_size 125M
99+
azure_kusto_buffer_key kusto1
100+
buffer_file_delete_early Off
101+
unify_tag On
102+
use_imds Off
103+
buffer_dir /var/log/
104+
store_dir_limit_size 16GB
105+
blob_uri_length 128
106+
scheduler_max_retries 3
107+
delete_on_max_upload_error Off
108+
io_timeout 60s
83109
```
84110

85111
## Troubleshooting

0 commit comments

Comments
 (0)