Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 27 additions & 29 deletions pipeline/outputs/azure_logs_ingestion.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,42 @@
---
description: 'Send logs to Azure Log Analytics using Logs Ingestion API with DCE and DCR'
description: Send logs to Azure Log Analytics using Logs Ingestion API
---

# Azure Logs Ingestion API

Azure Logs Ingestion plugin allows you to ingest your records using [Logs Ingestion API in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) to supported [Azure tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) or to [custom tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/create-custom-table#create-a-custom-table) that you create.
Azure Logs Ingestion plugin lets you ingest your records using [Logs Ingestion API in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) to supported [Azure tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) or to [custom tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/create-custom-table#create-a-custom-table) that you create.

The Logs ingestion API requires the following components:

- A Data Collection Endpoint (DCE)
- A Data Collection Rule (DCR) and
- A Log Analytics Workspace

> Note: According to [this document](https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/azure-monitor/logs/logs-ingestion-api-overview.md#components), all resources should be in the same region.
To visualize the basic logs ingestion operation, see the following image:

To visualize basic Logs Ingestion operation, see the following image:
![Log ingestion overview](../../.gitbook/assets/azure-logs-ingestion-overview.png)

![](../../.gitbook/assets/azure-logs-ingestion-overview.png)

To get more details about how to set up these components, please refer to the following documentations:
To get more details about how to set up these components, refer to the following documentation:

- [Azure Logs Ingestion API](https://docs.microsoft.com/en-us/azure/log-analytics/)
- [Send data to Azure Monitor Logs with Logs ingestion API (setup DCE, DCR and Log Analytics)](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal)

## Configuration Parameters
## Configuration parameters

| Key | Description | Default |
| :------------ | :------------------------- | :------ |
| tenant\_id | _Required_ - The tenant ID of the AAD application. ||
| client\_id | _Required_ - The client ID of the AAD application. ||
| client\_secret| _Required_ - The client secret of the AAD application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). ||
| dce\_url | _Required_ - Data Collection Endpoint(DCE) URL. ||
| dcr\_id | _Required_ - Data Collection Rule (DCR) immutable ID (see [this document](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal#collect-information-from-the-dcr) to collect the immutable id) ||
| table\_name | _Required_ - The name of the custom log table (include the `_CL` suffix as well if applicable) ||
| time\_key | _Optional_ - Specify the key name where the timestamp will be stored. | `@timestamp` |
| time\_generated | _Optional_ - If enabled, will generate a timestamp and append it to JSON. The key name is set by the 'time_key' parameter. | `true` |
| compress | _Optional_ - Enable HTTP payload gzip compression. | `true` |
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |

## Getting Started
| `tenant_id` | The tenant ID of the Azure Active Directory (AAD) application. | _none_ |
| `client_id` | The client ID of the AAD application. | _none_ |
| `client_secret`| The client secret of the AAD application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). | _none_ |
| `dce_url` | Data Collection Endpoint(DCE) URL. | _none_ |
| `dcr_id` | Data Collection Rule (DCR) [immutable ID](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal#collect-information-from-the-dcr). | _none_ |
| `table_name` | The name of the custom log table (include the `_CL` suffix as well if applicable) | _none_ |
| `time_key` | Optional. Specify the key name where the timestamp will be stored. | `@timestamp` |
| `time_generated` | Optional. If enabled, will generate a timestamp and append it to JSON. The key name is set by the `time_key` parameter. | `true` |
| `compress` | Optional. Enable HTTP payload gzip compression. | `true` |
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` |

## Get started

To send records into an Azure Log Analytics using Logs Ingestion API the following resources needs to be created:

Expand All @@ -47,11 +45,11 @@ To send records into an Azure Log Analytics using Logs Ingestion API the followi
- Either an [Azure tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) or [custom tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/create-custom-table#create-a-custom-table)
- An app registration with client secrets (for DCR access).

You can follow [this guideline](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal) to set up the DCE, DCR, app registration and a custom table.
Follow [this guideline](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal) to set up the DCE, DCR, app registration and a custom table.

### Configuration File
### Configuration file

Use this configuration to quickly get started:
Use this configuration file to get started:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand All @@ -62,9 +60,9 @@ pipeline:
- name: tail
path: /path/to/your/sample.log
tag: sample
key: RawData
# Or use other plugins
key: RawData

# Or use other plugins
#- name: cpu
# tag: sample

Expand All @@ -73,12 +71,12 @@ pipeline:
match: sample
# Add a json key named "Application":"fb_log"
add: Application fb_log

outputs:
# Enable this section to see your json-log format
#- name: stdout
# match: '*'

- name: azure_logs_ingestion
match: sample
client_id: XXXXXXXX-xxxx-yyyy-zzzz-xxxxyyyyzzzzxyzz
Expand Down Expand Up @@ -135,4 +133,4 @@ pipeline:
{% endtab %}
{% endtabs %}

Set up your DCR transformation accordingly based on the json output from fluent-bit's pipeline (input, parser, filter, output).
Set up your DCR transformation based on the JSON output from the Fluent Bit pipeline (input, parser, filter, output).
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Headings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ exceptions:
- API
- APIs
- Azure
- Azure Logs Ingestion API
- BuildKite
- CircleCI
- CLI
Expand Down