You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You can useincludeandexcludeprocessing rules to specify what data is sent toSumo Logic using OpenTelemetry Collector. Internally these will use [filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor) to get the data filtered.
10
+
You can useincludeandexcludeprocessing rules to define which data is sent toSumo Logic using the OpenTelemetry Collector. These rules internally utilize the [filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor)to filter the data.
17
11
18
-
* An exclude rule functions as a denylist filter where the matching data is not sent to Sumo Logic.
19
-
* An include rule functions as an allowlist filter where only matching data is sent to Sumo Logic.
12
+
* An exclude rule functions as a denylist filter, ensuring that matching data is not sent to Sumo Logic.
13
+
* An include rule functions as an allowlist filter, ensuring that only matching data is sent to Sumo Logic.
20
14
21
-
As a best practice, specify these rules to match the lesser volume of data.
15
+
As a best practice, configure these rules to filter the smaller volume of data for optimal performance:
22
16
23
-
* If you want to **collect the majority of data** from a source template, provide**exclude** rules to match (filter out) the lesser volume of data.
24
-
* If you want to **collect a small set of data** from a source template, provide**include** rules to match (filter in) the lesser volume of data.
17
+
* If you want to **collect the majority of data** from a source template, use**exclude** rules to match (filter out) the lesser volume of data.
18
+
* If you want to **collect a small set of data** from a source template, use**include** rules to match (filter in) the lesser volume of data.
25
19
26
-
For example, to include only messages coming from a Windows Event log with ID `8015`, you can add a Logs Filter to the source template and select the **Type** of the filter as "Include message that match", and can use the following filter regular expression:
20
+
For example, to include only messages from a Windows Event log with ID `8015`, you can add a Logs Filter to the source template. Select the **Type** of the filter as "Include messages that match" and use the following filter regular expression:
27
21
28
22
```
29
23
.*"id":8015.*
@@ -33,10 +27,10 @@ For example, to include only messages coming from a Windows Event log with ID `8
33
27
34
28
## Rules and limitations
35
29
36
-
When writing regular expression rules, you must follow these rules:
30
+
When creating regular expression rules, adhere to the following guidelines:
37
31
38
-
* Your rule must be [RE2 compliant](https://github.com/google/re2/wiki/Syntax).
39
-
* If your rule matches *only a section* of the log line, the full log line will be matched.
40
-
* For *singleline messages*, it is not mandatory to prefix and suffix the regex expression with `.\*`.
41
-
*Exclude rulestake priority over include rules. Include rules are processed first. However, if an exclude rule matches data that matched the include rule filter, the data is excluded.
42
-
* If two or more rules are listed, the assumed Boolean operator is `OR`.
32
+
- Your rule must comply with [RE2 syntax](https://github.com/google/re2/wiki/Syntax).
33
+
- If your rule matches *any part* of a log line, the entire log line will be matched.
34
+
- For *single-line messages*, it is not necessary to prefix or suffix the regex with `.*`.
35
+
-*Exclude rules*take precedence over *include rules*. Include rules are processed first, but if an exclude rule matches data that also matches the include rule, the data will be excluded.
36
+
- When multiple rules are listed, the assumed Boolean operator is `OR`.
This document only support masking logs for Windows source template. Refer to [Mask Rules](mask-rules.md) to mask logs for other source template.
9
+
This document supports masking logs specifically for our [Windows source template](/docs/send-data/opentelemetry-collector/remote-management/source-templates/windows). For other source templates, refer to [Mask Rules](mask-rules.md).
15
10
:::
16
11
17
-
A mask rule is a type of processing rule that hides irrelevant or sensitive information from logs before they are ingested. When you create a mask rule, the selected key will have its value matched against a regex pattern, which will then be replaced with a mask string before being sent to Sumo Logic. You can provide a custom mask string or use the default string, `"#####"`.
12
+
A mask rule is a type of processing rule that hides irrelevant or sensitive information from logs before they are ingested. When you create a mask rule:
13
+
14
+
* The selected key’s value is matched against a regular expression (regex).
15
+
* The matching portion is replaced with a mask string before being sent to Sumo Logic.
16
+
* You can provide a custom mask string or use the default mask string: `"#####"`.
17
+
18
+
Masking is an effective method for reducing overall ingestion volume. Ingestion volume is calculated after applying the mask filter. If masking reduces the log size, the smaller size will be considered against the ingestion limits.
18
19
19
-
Ingestion volume is calculated after applying the mask filter. If masking reduces the log size, the smaller size will be considered against the ingestion limits. Masking is an effective method for reducing overall ingestion volume.
20
+
## Masking inputs
20
21
21
22
To mask specific fields in the Windows Event Log, the following inputs are required:
22
-
-**Key**. This should point to the key in the Windows Event Log for which the value needs to be masked. This key can be nested, with each level separated by a dot(.). For example, `provider.guid`.
23
-
-**Regex**. This identifies the part of the string value that needs to be masked.
24
-
-**Replacement**. This is to get the string that will be substituted in place of the string that was selected through the regex expression.
23
+
-**Key**. This should point to the key in the Windows Event Log for which the value needs to be masked. This key can be nested, with each level separated by a dot (`.`). For example, `provider.guid`.
24
+
-**Regex**. This pattern identifies the part of the string value that needs to be masked.
25
+
-**Replacement**. The string to substitute for the matching portion identified by the regex.
25
26
26
27
:::important
27
28
Any masking expression should be tested and verified with a sample source file before applying it to your production logs.
28
29
:::
29
30
31
+
## Examples
32
+
33
+
### Masking numbers in a nested field
34
+
30
35
For example, to mask numbers inside `guid` under `provider` field from this log:
31
36
32
37
```
@@ -89,8 +94,7 @@ You could use the following masking expression input:
89
94
90
95
Using the above masking options would provide the following result:
91
96
92
-
```
93
-
{
97
+
`{
94
98
"record_id": 163054,
95
99
"channel": "Security",
96
100
"event_data": {
@@ -139,17 +143,16 @@ Using the above masking options would provide the following result:
139
143
"id": 4798
140
144
},
141
145
"level": "Information"
142
-
}
143
-
```
146
+
}`
144
147
145
148
:::note
146
-
-For masking, we use the [replace_pattern](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md#replace_pattern) OTTL function. In this function:
147
-
-$ must be escaped as $$ to bypass environment variable substitution logic.
148
-
-To input a literal $, use $$$.
149
+
-Masking utilizes the [replace_pattern](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md#replace_pattern) OTTL function. In this function:
150
+
-Escape `$`as `$$` to bypass environment variable substitution logic.
151
+
-Use `$$$` to include a literal `$`.
149
152
- When masking strings containing special characters like double quotes (`"`) and backslashes (`\`), these characters will be escaped by a backslash when masking the logs.
150
153
:::
151
154
152
155
## Limitations
153
156
154
157
- You can *only* mask the data which is a string in the Windows event log JSON.
155
-
- You cannot mask a value which is nested inside any array.
158
+
- You cannot mask a value that is nested inside any array.
This document do not support masking logs for Windows source template. Refer to [Mask Rules for Windows Source Template](mask-rules-windows.md) to mask logs for Windows source template.
8
+
This document does not cover masking logs for Windows source templates. For details on masking logs for Windows, refer to [Mask Rules for the Windows Source Template](mask-rules-windows.md).
14
9
:::
15
10
16
-
A mask rule is a type of processing rule that hides irrelevant or sensitive information from logs before ingestion. When you create amaskrule, whatever expression you choose to mask will be replaced with a mask string before it is sent toSumo Logic.You can provide a mask string, or use the default `"#####"`.
11
+
A mask rule is a processing rule that hides irrelevant or sensitive information from logs before they are ingested. When you create amaskrule, the selected expression will be replaced with a mask string before the data is sent toSumo Logic.You can either specify a custom mask string or use the default `"#####"`.
17
12
18
-
Ingestion volume is calculated after applying the mask filter. If the mask reduces the size of the log, the smaller size will be measured against ingestion limits. Masking is a good method for reducing overall ingestion volume.
13
+
Ingestion volume is calculated after applying the mask filter. If the mask reduces the size of the log, the smaller size will be measured against ingestion limits. Masking is an effective method to reduce overall ingestion volume.
19
14
20
15
For example, to mask the email address `[email protected]` from this log:
* Do not unnecessarily match on more of the log than needed. As seen in the previous example, avoid using overly broad expressions that could mask the entire log. This ensures that only the sensitive information is masked, not the whole log entry.
71
60
72
61
```
73
62
(?s).*auth"\s*:\s*"Basic\s*([^"]+)".*(?s)
74
63
```
75
64
76
-
* Make sure you do not specify a regular expression that matches a full log line. Doing so will result in the entire log line being masked.
65
+
* Avoid regular expressions that match an entire log line, as this will result in the entire line being masked.
77
66
78
-
* If you need to mask values on multiple lines, use single-line modifiers (?s). For example:
67
+
* To mask values spanning multiple lines, use the single-line modifier `(?s)`. For example:
79
68
80
69
```
81
70
auth=User\:(.*(?s).*session=.*?)\]
82
71
```
83
72
84
73
:::note
85
-
- For masking, we use the [replace_pattern](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md#replace_pattern) OTTL function. In this function:
86
-
- $ must be escaped as $$ to bypass environment variable substitution logic.
87
-
- To input a literal $, use $$$.
74
+
- Masking utilizes the [replace_pattern](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md#replace_pattern) OTTL function. In this function:
75
+
- Escape `$` as `$$` to bypass environment variable substitution logic.
76
+
- Use `$$$` to include a literal `$`.
88
77
- When masking strings containing special characters like double quotes (`"`) and backslashes (`\`), these characters will be escaped by a backslash when masking the logs.
89
78
:::
90
79
@@ -98,6 +87,8 @@ Any masking expression should be tested and verified with a sample source file b
98
87
99
88
You can mask credit card numbers from log messages using a regular expression within a mask rule. Once masked with a known string, you can then perform a search for that string within your logs to detect if credit card numbers may be leaking into your log files.
100
89
90
+
To mask credit card numbers in logs, you can use a masking filter with the following regular expression:
91
+
101
92
The following regular expression can be used within a masking filter to mask American Express, Visa (16 digit only), Mastercard, and Discover credit card numbers:
102
93
103
94
```
@@ -108,7 +99,7 @@ This regular expression covers instances where the number includes dashes, space
0 commit comments