Skip to content

Added missing YAML examples for Azure output plugins collection. Fixes #1897. #1898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jul 14, 2025
Merged

Added missing YAML examples for Azure output plugins collection. Fixes #1897. #1898

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a3100aa
Adding YAML exampels and sandardizing shell usage. Part of issue #1897.
eschabell Jul 11, 2025
2f4efde
Adding YAML exampels and sandardizing shell usage for Azure Data Expl…
eschabell Jul 11, 2025
c28ec75
Adding YAML exampels and sandardizing shell usage for Azure Log Analy…
eschabell Jul 11, 2025
53119fc
Adding YAML exampels and sandardizing shell usage for Azure Log Inges…
eschabell Jul 11, 2025
7d72879
Update pipeline/outputs/azure_kusto.md
alexakreizinger Jul 11, 2025
9a219fa
Adjusted azure blog code examples to use 2 space indents. Part of iss…
eschabell Jul 12, 2025
79a0c47
Merge branch 'erics-azure-outputs-yaml' of github.com:eschabell/fluen…
eschabell Jul 12, 2025
543c9e8
Adjusted azure data explorer code examples to use 2 space indents. Pa…
eschabell Jul 12, 2025
3be0edf
Adjusted azure log analytics code examples to use 2 space indents. Pa…
eschabell Jul 12, 2025
7a4e178
Adjusted azure logs ingestion api code examples to use 2 space indent…
eschabell Jul 12, 2025
0e97081
Fixed a fake shared key and acount name in azure blob output plugin d…
eschabell Jul 14, 2025
4313344
Merge branch 'master' into erics-azure-outputs-yaml
eschabell Jul 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 55 additions & 14 deletions pipeline/outputs/azure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ description: 'Send logs, metrics to Azure Log Analytics'

# Azure Log Analytics

![](../../.gitbook/assets/image%20%287%29.png)

Azure output plugin allows to ingest your records into [Azure Log Analytics](https://azure.microsoft.com/en-us/services/log-analytics/) service.

To get more details about how to setup Azure Log Analytics, please refer to the following documentation: [Azure Log Analytics](https://docs.microsoft.com/en-us/azure/log-analytics/)
Expand All @@ -30,35 +28,78 @@ In order to insert records into an Azure Log Analytics instance, you can run the

The **azure** plugin, can read the parameters from the command line in two ways, through the **-p** argument \(property\), e.g:

```text
```shell
fluent-bit -i cpu -o azure -p customer_id=abc -p shared_key=def -m '*' -f 1
```

### Configuration File

In your main configuration file append the following _Input_ & _Output_ sections:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: cpu

outputs:
- name: azure
match: '*'
customer_id: abc
shared_key: def
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name cpu
Name cpu

[OUTPUT]
Name azure
Match *
Customer_ID abc
Shared_Key def
Name azure
Match *
Customer_ID abc
Shared_Key def
```

{% endtab %}
{% endtabs %}

Another example using the `Log_Type_Key` with [record-accessor](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/record-accessor), which will read the table name (or event type) dynamically from kubernetes label `app`, instead of `Log_Type`:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: cpu

outputs:
- name: azure
match: '*'
log_type_key: $kubernetes['labels']['app']
customer_id: abc
shared_key: def
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name cpu
Name cpu

[OUTPUT]
Name azure
Match *
Log_Type_Key $kubernetes['labels']['app']
Customer_ID abc
Shared_Key def
Name azure
Match *
Log_Type_Key $kubernetes['labels']['app']
Customer_ID abc
Shared_Key def
```

{% endtab %}
{% endtabs %}
195 changes: 131 additions & 64 deletions pipeline/outputs/azure_blob.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,34 +16,34 @@ Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial \(Video\)

Fluent Bit exposes the following configuration properties.

| Key | Description | Default |
| :--- | :--- | :--- |
| `account_name` | Azure Storage account name. | _none_ |
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
| `container_name` | Name of the container that will contain the blobs. | _none_ |
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ |
| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ |
| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ |
| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` |
| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ |
| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` |
| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` |
| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` |
| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` |
| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` |
| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` |
| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` |
| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` |
| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` |
| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` |
| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` |
| `io_timeout` | HTTP IO timeout. | `60s` |
| Key | Description | Default |
| :--------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- |
| `account_name` | Azure Storage account name. | _none_ |
| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` |
| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ |
| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ |
| `container_name` | Name of the container that will contain the blobs. | _none_ |
| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` |
| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `on` |
| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ |
| `compress` | Sets payload compression in network transfer. Supported value: `gzip` | _none_ |
| `compress_blob` | Enables GZIP compression in the final `blockblob` file. This option isn't compatible when `blob_type` = `appendblob`. | _none_ |
| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `off` |
| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ |
| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |
| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` |
| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` |
| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and ago haven't reached `upload_file_size` limit yet. | `30m` |
| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` |
| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` |
| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` |
| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` |
| `blob_uri_length` | Set the length of generated blob URI before ingesting to Azure Kusto. | `64` |
| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` |
| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` |
| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` |
| `io_timeout` | HTTP IO timeout. | `60s` |

## Get started

Expand All @@ -53,28 +53,60 @@ Fluent Bit can deliver records to the official service or an emulator.

The following configuration example generates a random message with a custom tag:

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info

pipeline:
inputs:
- name: dummy
dummy: '{"name": "Fluent Bit", "year": 2020}'
samples: 1
tag: var.log.containers.app-default-96cbdef2340.log

outputs:
- name: azure_blog
match: "*"
account_name: YOUR_ACCOUNT_NAME
shared_key: YOUR_SHARED_KEY
path: kubernetes
container_name: logs
auto_create_container: on
tls: on
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_level info
flush 1
log_level info

[INPUT]
name dummy
dummy {"name": "Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log
name dummy
dummy {"name": "Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
shared_key YOUR_SHARED_KEY
path kubernetes
container_name logs
auto_create_container on
tls on
name azure_blob
match *
account_name YOUR_ACCOUNT_NAME
shared_key YOUR_SHARED_KEY
path kubernetes
container_name logs
auto_create_container on
tls on
```

{% endtab %}
{% endtabs %}

After you run the configuration file, you will be able to query the data using the Azure Storage Explorer. The example generates the following content in the explorer:

![Azure Blob](../../.gitbook/assets/azure_blob.png)
Expand All @@ -85,13 +117,13 @@ After you run the configuration file, you will be able to query the data using t

1. Install Azurite using `npm`:

```bash
```shell
npm install -g azurite
```

1. Run the service:

```bash
```shell
azurite
```

Expand All @@ -108,34 +140,69 @@ After you run the configuration file, you will be able to query the data using t

[Azurite](https://github.com/Azure/Azurite) comes with a default `account_name` and `shared_key`. Instead of the defaults, be sure to use the specific values provided in the following example:

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info

pipeline:
inputs:
- name: dummy
dummy: '{"name": "Fluent Bit", "year": 2020}'
samples: 1
tag: var.log.containers.app-default-96cbdef2340.log

outputs:
- name: azure_blog
match: "*"
account_name: INSERT_ACCOUNT_NAME
shared_key: INSERT_SHARED_KEY
path: kubernetes
container_name: logs
auto_create_container: on
tls: off
emulator_mode: on
endpoint: http://127.0.0.1:10000
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_level info
flush 1
log_level info

[INPUT]
name dummy
dummy {"name": "Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log
name dummy
dummy {"name": "Fluent Bit", "year": 2020}
samples 1
tag var.log.containers.app-default-96cbdef2340.log

[OUTPUT]
name azure_blob
match *
account_name devstoreaccount1
shared_key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
path kubernetes
container_name logs
auto_create_container on
tls off
emulator_mode on
endpoint http://127.0.0.1:10000
name azure_blob
match *
account_name INSERT_ACCOUNT_NAME
shared_key INSERT_SHARED_KEY
path kubernetes
container_name logs
auto_create_container on
tls off
emulator_mode on
endpoint http://127.0.0.1:10000
```

{% endtab %}
{% endtabs %}

After running the Fluent Bit configuration, you will see the data flowing into Azurite:

```text
```shell
$ azurite

Azurite Blob service is starting at http://127.0.0.1:10000
Azurite Blob service is successfully listening at http://127.0.0.1:10000
Azurite Queue service is starting at http://127.0.0.1:10001
Expand All @@ -145,4 +212,4 @@ Azurite Queue service is successfully listening at http://127.0.0.1:10001
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -
127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -
127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -
```
```
Loading