Skip to content

Commit cd31571

Browse files
eschabelleschabell
authored
Merge pull request #1898 from eschabell/erics-azure-outputs-yaml
Added missing YAML examples for Azure output plugins collection. Fixes #1897.
2 parents f419eab + 4313344 commit cd31571

File tree

4 files changed

+379
-177
lines changed

4 files changed

+379
-177
lines changed

‎pipeline/outputs/azure.md

Lines changed: 55 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,6 @@ description: 'Send logs, metrics to Azure Log Analytics'
44

55
# Azure Log Analytics
66

7-
![](../../.gitbook/assets/image%20%287%29.png)
8-
97
Azure output plugin allows to ingest your records into [Azure Log Analytics](https://azure.microsoft.com/en-us/services/log-analytics/) service.
108

119
To get more details about how to setup Azure Log Analytics, please refer to the following documentation: [Azure Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/)
@@ -30,35 +28,78 @@ In order to insert records into an Azure Log Analytics instance, you can run the
3028

3129
The **azure** plugin, can read the parameters from the command line in two ways, through the **-p** argument \(property\), e.g:
3230

33-
```text
31+
```shell
3432
fluent-bit -i cpu -o azure -p customer_id=abc -p shared_key=def -m '*' -f 1
3533
```
3634

3735
### Configuration File
3836

3937
In your main configuration file append the following _Input_ & _Output_ sections:
4038

39+
{% tabs %}
40+
{% tab title="fluent-bit.yaml" %}
41+
42+
```yaml
43+
pipeline:
44+
inputs:
45+
- name: cpu
46+
47+
outputs:
48+
- name: azure
49+
match: '*'
50+
customer_id: abc
51+
shared_key: def
52+
```
53+
54+
{% endtab %}
55+
{% tab title="fluent-bit.conf" %}
56+
4157
```text
4258
[INPUT]
43-
Name cpu
59+
Name cpu
4460

4561
[OUTPUT]
46-
Name azure
47-
Match *
48-
Customer_ID abc
49-
Shared_Key def
62+
Name azure
63+
Match *
64+
Customer_ID abc
65+
Shared_Key def
5066
```
5167

68+
{% endtab %}
69+
{% endtabs %}
70+
5271
Another example using the `Log_Type_Key` with [record-accessor](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/record-accessor), which will read the table name (or event type) dynamically from kubernetes label `app`, instead of `Log_Type`:
5372

73+
{% tabs %}
74+
{% tab title="fluent-bit.yaml" %}
75+
76+
```yaml
77+
pipeline:
78+
inputs:
79+
- name: cpu
80+
81+
outputs:
82+
- name: azure
83+
match: '*'
84+
log_type_key: $kubernetes['labels']['app']
85+
customer_id: abc
86+
shared_key: def
87+
```
88+
89+
{% endtab %}
90+
{% tab title="fluent-bit.conf" %}
91+
5492
```text
5593
[INPUT]
56-
Name cpu
94+
Name cpu
5795

5896
[OUTPUT]
59-
Name azure
60-
Match *
61-
Log_Type_Key $kubernetes['labels']['app']
62-
Customer_ID abc
63-
Shared_Key def
97+
Name azure
98+
Match *
99+
Log_Type_Key $kubernetes['labels']['app']
100+
Customer_ID abc
101+
Shared_Key def
64102
```
103+
104+
{% endtab %}
105+
{% endtabs %}

‎pipeline/outputs/azure_blob.md

Lines changed: 131 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -16,34 +16,34 @@ Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial \(Video\)
1616

1717
Fluent Bit exposes the following configuration properties.
1818

19-
| Key | Description | Default |
20-
| :--- | :--- | :--- |
21-
| `account_name` | Azure Storage account name. | _none_ |
22-
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
23-
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
24-
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
25-
| `container_name` | Name of the container that will contain the blobs. | _none_ |
26-
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
27-
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
28-
| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ |
29-
| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ |
30-
| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ |
31-
| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` |
32-
| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ |
33-
| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` |
34-
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
35-
| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` |
36-
| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` |
37-
| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` |
38-
| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` |
39-
| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` |
40-
| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` |
41-
| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` |
42-
| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` |
43-
| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` |
44-
| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` |
45-
| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` |
46-
| `io_timeout` | HTTP IO timeout. | `60s` |
19+
| Key | Description | Default |
20+
| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- |
21+
| `account_name` | Azure Storage account name. | _none_ |
22+
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
23+
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
24+
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
25+
| `container_name` | Name of the container that will contain the blobs. | _none_ |
26+
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
27+
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
28+
| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ |
29+
| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ |
30+
| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ |
31+
| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` |
32+
| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ |
33+
| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` |
34+
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
35+
| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` |
36+
| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` |
37+
| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` |
38+
| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` |
39+
| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` |
40+
| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` |
41+
| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` |
42+
| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` |
43+
| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` |
44+
| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` |
45+
| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` |
46+
| `io_timeout` | HTTP IO timeout. | `60s` |
4747

4848
## Get started
4949

@@ -53,28 +53,60 @@ Fluent Bit can deliver records to the official service or an emulator.
5353

5454
The following configuration example generates a random message with a custom tag:
5555

56-
```python
56+
{% tabs %}
57+
{% tab title="fluent-bit.yaml" %}
58+
59+
```yaml
60+
service:
61+
flush: 1
62+
log_level: info
63+
64+
pipeline:
65+
inputs:
66+
- name: dummy
67+
dummy: '{"name": "Fluent Bit", "year": 2020}'
68+
samples: 1
69+
tag: var.log.containers.app-default-96cbdef2340.log
70+
71+
outputs:
72+
- name: azure_blog
73+
match: "*"
74+
account_name: YOUR_ACCOUNT_NAME
75+
shared_key: YOUR_SHARED_KEY
76+
path: kubernetes
77+
container_name: logs
78+
auto_create_container: on
79+
tls: on
80+
```
81+
82+
{% endtab %}
83+
{% tab title="fluent-bit.conf" %}
84+
85+
```text
5786
[SERVICE]
58-
flush 1
59-
log_level info
87+
flush 1
88+
log_level info
6089

6190
[INPUT]
62-
name dummy
63-
dummy {"name": "Fluent Bit", "year": 2020}
64-
samples 1
65-
tag var.log.containers.app-default-96cbdef2340.log
91+
name dummy
92+
dummy {"name": "Fluent Bit", "year": 2020}
93+
samples 1
94+
tag var.log.containers.app-default-96cbdef2340.log
6695

6796
[OUTPUT]
68-
name azure_blob
69-
match *
70-
account_name YOUR_ACCOUNT_NAME
71-
shared_key YOUR_SHARED_KEY
72-
path kubernetes
73-
container_name logs
74-
auto_create_container on
75-
tls on
97+
name azure_blob
98+
match *
99+
account_name YOUR_ACCOUNT_NAME
100+
shared_key YOUR_SHARED_KEY
101+
path kubernetes
102+
container_name logs
103+
auto_create_container on
104+
tls on
76105
```
77106

107+
{% endtab %}
108+
{% endtabs %}
109+
78110
After you run the configuration file, you will be able to query the data using the Azure Storage Explorer. The example generates the following content in the explorer:
79111

80112
![Azure Blob](../../.gitbook/assets/azure_blob.png)
@@ -85,13 +117,13 @@ After you run the configuration file, you will be able to query the data using t
85117

86118
1. Install Azurite using `npm`:
87119

88-
```bash
120+
```shell
89121
npm install -g azurite
90122
```
91123

92124
1. Run the service:
93125

94-
```bash
126+
```shell
95127
azurite
96128
```
97129

@@ -108,34 +140,69 @@ After you run the configuration file, you will be able to query the data using t
108140

109141
[Azurite](https://github.com/Azure/Azurite) comes with a default `account_name` and `shared_key`. Instead of the defaults, be sure to use the specific values provided in the following example:
110142

111-
```python
143+
{% tabs %}
144+
{% tab title="fluent-bit.yaml" %}
145+
146+
```yaml
147+
service:
148+
flush: 1
149+
log_level: info
150+
151+
pipeline:
152+
inputs:
153+
- name: dummy
154+
dummy: '{"name": "Fluent Bit", "year": 2020}'
155+
samples: 1
156+
tag: var.log.containers.app-default-96cbdef2340.log
157+
158+
outputs:
159+
- name: azure_blog
160+
match: "*"
161+
account_name: INSERT_ACCOUNT_NAME
162+
shared_key: INSERT_SHARED_KEY
163+
path: kubernetes
164+
container_name: logs
165+
auto_create_container: on
166+
tls: off
167+
emulator_mode: on
168+
endpoint: http://127.0.0.1:10000
169+
```
170+
171+
{% endtab %}
172+
{% tab title="fluent-bit.conf" %}
173+
174+
```text
112175
[SERVICE]
113-
flush 1
114-
log_level info
176+
flush 1
177+
log_level info
115178

116179
[INPUT]
117-
name dummy
118-
dummy {"name": "Fluent Bit", "year": 2020}
119-
samples 1
120-
tag var.log.containers.app-default-96cbdef2340.log
180+
name dummy
181+
dummy {"name": "Fluent Bit", "year": 2020}
182+
samples 1
183+
tag var.log.containers.app-default-96cbdef2340.log
121184

122185
[OUTPUT]
123-
name azure_blob
124-
match *
125-
account_name devstoreaccount1
126-
shared_key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
127-
path kubernetes
128-
container_name logs
129-
auto_create_container on
130-
tls off
131-
emulator_mode on
132-
endpoint http://127.0.0.1:10000
186+
name azure_blob
187+
match *
188+
account_name INSERT_ACCOUNT_NAME
189+
shared_key INSERT_SHARED_KEY
190+
path kubernetes
191+
container_name logs
192+
auto_create_container on
193+
tls off
194+
emulator_mode on
195+
endpoint http://127.0.0.1:10000
133196
```
134197

198+
{% endtab %}
199+
{% endtabs %}
200+
135201
After running the Fluent Bit configuration, you will see the data flowing into Azurite:
136202

137-
```text
203+
```shell
138204
$ azurite
205+
139206
Azurite Blob service is starting at http://127.0.0.1:10000
140207
Azurite Blob service is successfully listening at http://127.0.0.1:10000
141208
Azurite Queue service is starting at http://127.0.0.1:10001
@@ -145,4 +212,4 @@ Azurite Queue service is successfully listening at http://127.0.0.1:10001
145212
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -
146213
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -
147214
127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -
148-
```
215+
```

0 commit comments

Comments
 (0)