You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use the Parse JSON processor to parse the `message` field so the `message` field has all the attributes within a nested object.
36
+
37
+
{{< img src="observability_pipelines/processors/parse-json-example.png" alt="The parse json processor with message as the field to parse on" style="width:60%;" >}}
38
+
39
+
This output contains the `message` field with the parsed JSON:
40
+
41
+
```json
42
+
{
43
+
"foo": "bar",
44
+
"team": "my-team",
45
+
"message": {
46
+
"action": "login",
47
+
"ip_address": "192.168.1.100",
48
+
"level": "info",
49
+
"service": "user-service",
50
+
"success": true,
51
+
"timestamp": "2024-01-15T10:30:00Z",
52
+
"user_id": "12345"
53
+
}
54
+
"app_id":"streaming-services",
55
+
"ddtags": [
56
+
"kube_service:my-service",
57
+
"k8_deployment :your-host"
58
+
]
59
+
}
60
+
```
61
+
62
+
## Setup
63
+
64
+
To set up this processor:
65
+
1. Define a **filter query**. Only logs that match the specified [filter query](#filter-query-syntax) are processed. All logs, regardless of whether they do or do not match the filter query, are sent to the next step in the pipeline.
66
+
2. Enter the name of the field you want to parse JSON on.<br>**Note**: The parsed JSON overwrites what was originally contained in the field.
This processor parses Extensible Markup Language (XML) so the data can be processed and sent to different destinations. XML is a log format used to store and transport structured data. It is organized in a tree-like structure to represent nested information and uses tags and attributes to define the data. For example, this is XML data using only tags (`<recipe>`,`<type>`, and `<name>`) and no attributes:
22
+
23
+
```xml
24
+
<recipe>
25
+
<type>pasta</type>
26
+
<name>Carbonara</name>
27
+
</recipe>
28
+
```
29
+
30
+
This is an XML example where the tag `recipe` has the attribute `type`:
31
+
32
+
```xml
33
+
<recipe>
34
+
<recipetype="pasta">
35
+
<name>Carbonara</name>
36
+
</recipe>
37
+
```
38
+
39
+
The following image shows a Windows Event 4625 log in XML, next to the same log parsed and output in JSON. By parsing the XML log, the size of the log event was reduced by approximately 30%.
40
+
41
+
{{< img src="observability_pipelines/processors/xml-side-by-side.png" alt="The XML log and the resulting parsed log in JSON" style="width:80%;" >}}
42
+
43
+
## Setup
44
+
45
+
To set up this processor:
46
+
47
+
1. Define a filter query. Only logs that match the specified filter query are processed. All logs, regardless of whether they match the filter query, are sent to the next step in the pipeline.
48
+
1. Enter the path to the log field on which you want to parse XML. Use the path notation `<OUTER_FIELD>.<INNER_FIELD>` to match subfields. See the [Path notation example](#path-notation-example-parse-xml) below.
49
+
1. Optionally, in the `Enter text key` field, input the key name to use for the text node when XML attributes are appended. See the [text key example](#text-key-example). If the field is left empty, `value` is used as the key name.
50
+
1. Optionally, select `Always use text key` if you want to store text inside an object using the text key even when no attributes exist.
51
+
1. Optionally, toggle `Include XML attributes` on if you want to include XML attributes. You can then choose to add the attribute prefix you want to use. See [attribute prefix example](#attribute-prefix-example). If the field is left empty, the original attribute key is used.
52
+
1. Optionally, select if you want to convert data types into numbers, Booleans, or nulls.
53
+
- If **Numbers** is selected, numbers are parsed as integers and floats.
54
+
- If **Booleans** is selected, `true` and `false` are parsed as Booleans.
55
+
- If **Nulls** is selected, the string `null` is parsed as null.
56
+
57
+
##### Path notation example {#path-notation-example-parse-xml}
58
+
59
+
For the following message structure:
60
+
61
+
```json
62
+
{
63
+
"outer_key": {
64
+
"inner_key": "inner_value",
65
+
"a": {
66
+
"double_inner_key": "double_inner_value",
67
+
"b": "b value"
68
+
},
69
+
"c": "c value"
70
+
},
71
+
"d": "d value"
72
+
}
73
+
```
74
+
75
+
- Use `outer_key.inner_key` to see the key with the value `inner_value`.
76
+
- Use `outer_key.inner_key.double_inner_key` to see the key with the value `double_inner_value`.
77
+
78
+
##### Always use text key example
79
+
80
+
If **Always use text key** is selected, the text key is the default (`value`), and you have the following XML:
81
+
82
+
```xml
83
+
<recipe>
84
+
<recipetype="pasta">
85
+
<name>Carbonara</name>
86
+
</recipe>
87
+
```
88
+
89
+
The XML is converted to:
90
+
91
+
```json
92
+
{
93
+
"recipe": {
94
+
"type": "pasta",
95
+
"value": "Carbonara"
96
+
}
97
+
}
98
+
```
99
+
100
+
##### Text key example
101
+
102
+
If the key is `text` and you have the following XML:
103
+
104
+
```xml
105
+
<recipe>
106
+
<recipetype="pasta">
107
+
<name>Carbonara</name>
108
+
</recipe>
109
+
```
110
+
111
+
The XML is converted to:
112
+
113
+
```json
114
+
{
115
+
"recipe": {
116
+
"type": "pasta",
117
+
"text": "Carbonara"
118
+
}
119
+
}
120
+
```
121
+
122
+
##### Attribute prefix example
123
+
124
+
If you enable **Include XML attributes**, the attribute is added as a prefix to each XML attribute. For example, if the attribute prefix is `@` and you have the following XML:
Copy file name to clipboardExpand all lines: content/en/observability_pipelines/processors/quota.md
+78-1Lines changed: 78 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,6 +9,83 @@ products:
9
9
10
10
{{< product-availability >}}
11
11
12
-
{{% observability_pipelines/processors/quota %}}
12
+
## Overview
13
+
14
+
The quota processor measures the logging traffic for logs that match the filter you specify. When the configured daily quota is met inside the 24-hour rolling window, the processor can either keep or drop additional logs, or send them to a storage bucket. For example, you can configure this processor to drop new logs or trigger an alert without dropping logs after the processor has received 10 million events from a certain service in the last 24 hours.
15
+
16
+
You can also use field-based partitioning, such as `service`, `env`, `status`. Each unique fields uses a separate quota bucket with its own daily quota limit. See [Partition example](#partition-example) for more information.
17
+
18
+
**Note**: The pipeline uses the name of the quota to identify the same quota across multiple Remote Configuration deployments of the Worker.
19
+
20
+
### Limits
21
+
22
+
- Each pipeline can have up to 1000 buckets. If you need to increase the bucket limit, [contact support][5].
23
+
- The quota processor is synchronized across all Workers in a Datadog organization. For the synchronization, there is a default rate limit of 50 Workers per organization. When there are more than 50 Workers for an organization:
24
+
- The processor continues to run, but does not sync correctly with the other Workers, which can result in logs being sent after the quota limit has been reached.
25
+
- The Worker prints `Failed to sync quota state` errors.
26
+
-[Contact support][5] if you want to increase the default number of Workers per organization.
27
+
- The quota processor periodically synchronizes counts across Workers a few times per minute. The limit set on the processor can therefore be overshot, depending on the number of Workers and the logs throughput. Datadog recommends setting a limit that is at least one order of magnitude higher than the volume of logs that the processor is expected to receive per minute. You can use a throttle processor with the quota processor to control these short bursts by limiting the number of logs allowed per minute.
28
+
29
+
## Setup
30
+
31
+
To set up the quota processor:
32
+
1. Enter a name for the quota processor.
33
+
1. Define a **filter query**. Only logs that match the specified [filter query](#filter-query-syntax) are counted towards the daily limit.
34
+
- Logs that match the quota filter and are within the daily quota are sent to the next step in the pipeline.
35
+
- Logs that do not match the quota filter are sent to the next step of the pipeline.
36
+
1. In the **Unit for quota** dropdown menu, select if you want to measure the quota by the number of `Events` or by the `Volume` in bytes.
37
+
1. Set the daily quota limit and select the unit of magnitude for your desired quota.
38
+
1. Optional: Click **Add Field** if you want to set a quota on a specific service or region field.
39
+
1. Enter the field name you want to partition by. See the [Partition example](#partition-example) for more information.
40
+
1. Select the **Ignore when missing** if you want the quota applied only to events that match the partition. See the [Ignore when missing example](#example-for-the-ignore-when-missing-option) for more information.
41
+
1. Optional: Click **Overrides** if you want to set different quotas for the partitioned field.
42
+
- Click **Download as CSV** for an example of how to structure the CSV.
43
+
- Drag and drop your overrides CSV to upload it. You can also click **Browse** to select the file to upload it. See the [Overrides example](#overrides-example) for more information.
44
+
1. Click **Add Field** if you want to add another partition.
45
+
1. In the **When quota is met** dropdown menu, select if you want to **drop events**, **keep events**, or **send events to overflow destination**, when the quota has been met.
46
+
1. If you select **send events to overflow destination**, an overflow destination is added with the following cloud storage options: **Amazon S3**, **Azure Blob**, and **Google Cloud**.
47
+
1. Select the cloud storage you want to send overflow logs to. See the setup instructions for your cloud storage: [Amazon S3][2], [Azure Blob Storage][3], or [Google Cloud Storage][4].
48
+
49
+
#### Examples
50
+
51
+
##### Partition example
52
+
53
+
Use **Partition by** if you want to set a quota on a specific service or region. For example, if you want to set a quota for 10 events per day and group the events by the `service` field, enter `service` into the **Partition by** field.
54
+
55
+
##### Example for the "ignore when missing" option
56
+
57
+
Select **Ignore when missing** if you want the quota applied only to events that match the partition. For example, if the Worker receives the following set of events:
58
+
59
+
```
60
+
{"service":"a", "source":"foo", "message": "..."}
61
+
{"service":"b", "source":"bar", "message": "..."}
62
+
{"service":"b", "message": "..."}
63
+
{"source":"redis", "message": "..."}
64
+
{"message": "..."}
65
+
```
66
+
67
+
And the **Ignore when missing** is selected, then the Worker:
68
+
- creates a set for logs with `service:a` and `source:foo`
69
+
- creates a set for logs with `service:b` and `source:bar`
70
+
- ignores the last three events
71
+
72
+
The quota is applied to the two sets of logs and not to the last three events.
73
+
74
+
If the **Ignore when missing** is not selected, the quota is applied to all five events.
75
+
76
+
##### Overrides example
77
+
78
+
If you are partitioning by `service` and have two services: `a` and `b`, you can use overrides to apply different quotas for them. For example, if you want `service:a` to have a quota limit of 5,000 bytes and `service:b` to have a limit of 50 events, the override rules look like this:
Copy file name to clipboardExpand all lines: content/en/observability_pipelines/processors/reduce.md
+34-1Lines changed: 34 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,6 +9,39 @@ products:
9
9
10
10
{{< product-availability >}}
11
11
12
-
{{% observability_pipelines/processors/reduce %}}
12
+
## Overview
13
+
14
+
The reduce processor groups multiple log events into a single log, based on the fields specified and the merge strategies selected. Logs are grouped at 10-second intervals. After the interval has elapsed for the group, the reduced log for that group is sent to the next step in the pipeline.
15
+
16
+
## Setup
17
+
18
+
To set up the reduce processor:
19
+
1. Define a **filter query**. Only logs that match the specified [filter query](#filter-query-syntax) are processed. Reduced logs and logs that do not match the filter query are sent to the next step in the pipeline.
20
+
2. In the **Group By** section, enter the field you want to group the logs by.
21
+
3. Click **Add Group by Field** to add additional fields.
22
+
4. In the **Merge Strategy** section:
23
+
- In **On Field**, enter the name of the field you want to merge the logs on.
24
+
- Select the merge strategy in the **Apply** dropdown menu. This is the strategy used to combine events. See the following [Merge strategies](#merge-strategies) section for descriptions of the available strategies.
25
+
- Click **Add Merge Strategy** to add additional strategies.
26
+
27
+
##### Merge strategies
28
+
29
+
These are the available merge strategies for combining log events.
0 commit comments