Skip to content

Commit 57d260b

Browse files
efd6efd6
authored
varonis: add pre-processor option to allow ingestion of non-conformant CEF messages (#13822)
Add a config option to add "pre-processors" which will run before the "decode_cef" processor on the agent. CEF messages from some sources may be malformed by not following the CEF spec exactly, which can then cause problems in the "decode_cef" processor. Having a pre-processor will allow running scripts that can correct the malformed CEF messages, and allow decode_cef to process them correctly. New test sample provided in issue from user.
1 parent 2428fdf commit 57d260b

File tree

12 files changed

+140
-1
lines changed

12 files changed

+140
-1
lines changed

‎packages/varonis/_dev/build/docs/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ Users can configure the syslog server address in DatAlert so that alerts can be
2929

3030
This integration expects to use `External system default template (CEF)` for alert forwarding in Varonis DatAlert tool. In case any custom template is used, all the fields in `External system default template (CEF)` should also be present in custom template along with the other additional fields. Additional fields will be part of `varonis.logs` object and such fields will be indexed only if dynamic mapping is enabled in Elasticsearch.
3131

32+
## Pre-Processors
33+
34+
There are cases where incoming CEF messages do not follow the CEF specification exactly, and this can cause errors with
35+
message decoding. To work around this, there is an option for pre-processors, which are run before the CEF message is decoded.
36+
These can be used modify the message to follow the CEF specification correctly, which will allow proper decoding.
37+
38+
The pre-processors will modify the `message` field before CEF decoding is done on the agent, but the original, non-preprocessed,
39+
message will still be preserved in the `event.original` field when the agent sends the event.
40+
3241
## Logs reference
3342

3443
### varonis.logs

‎packages/varonis/_dev/deploy/docker/docker-compose.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,13 @@ services:
1010
volumes:
1111
- ./sample_logs:/sample_logs:ro
1212
command: log --start-signal=SIGHUP --delay=6s --addr elastic-agent:9031 -p=tcp /sample_logs/varonis.log
13+
varonis-udp-invalid-cef:
14+
image: docker.elastic.co/observability/stream:v0.15.0
15+
volumes:
16+
- ./sample_logs:/sample_logs:ro
17+
command: log --start-signal=SIGHUP --delay=6s --addr elastic-agent:9032 -p=udp /sample_logs/varonis_invalid_cef.log
18+
varonis-tcp-invalid-cef:
19+
image: docker.elastic.co/observability/stream:v0.15.0
20+
volumes:
21+
- ./sample_logs:/sample_logs:ro
22+
command: log --start-signal=SIGHUP --delay=6s --addr elastic-agent:9033 -p=tcp /sample_logs/varonis_invalid_cef.log

‎packages/varonis/_dev/deploy/docker/sample_logs/varonis_invalid_cef.log‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
<14>Apr 28 15:59:01 VARONISSRVR1 CEF:0|Varonis Inc.|DatAdvantage|8.7.1|5015|DS object permission added|3|rt=Apr 28 2025 15:01:42 cat=Alert cs2=Permission changes on OU cs2Label=RuleName cn1=142 cn1Label=RuleID end=Apr 28 2025 14:53:02 duser=contosofoo.com\Bourne Jr., Jason dhost=AD-contosofoo.com filePath=contosofoo.com\Home Office Site\Computers\Containered fname=Containered act=DS object permission added dvchost=Compute outcome=Success msg=Permissions were added to "Containered" for group "contosofoo.com\Group Admins-Region_200" cs3= cs3Label=AttachmentName cs4= cs4Label=ClientAccessType deviceCustomDate1= fileType= cs1= cs1Label=MailRecipient suser= cs5= cs5Label=MailboxAccessType cnt= cs6= cs6Label=ChangedPermissions oldFilePermission= filePermission=Write dpriv=contosofoo.com\Group Admins-Region_200 start=Apr 28 2025 14:53:02
2+
<14>Apr 28 15:59:02 VARONISSRVR1 CEF:0|Varonis Inc.|DatAdvantage|8.7.1|5015|DS object permission added|3|rt=Apr 28 2025 15:01:42 cat=Alert cs2=Permission changes on OU cs2Label=RuleName cn1=142 cn1Label=RuleID end=Apr 28 2025 14:53:04 duser=contosofoo.com\Bourne Jr., Jason dhost=AD-contosofoo.com filePath=contosofoo.com\Home Office Site\Computers\Containered fname=Containered act=DS object permission added dvchost=Compute outcome=Success msg=Permissions were added to "Containered" for group "contosofoo.com\Group Admins-Region_200" cs3= cs3Label=AttachmentName cs4= cs4Label=ClientAccessType deviceCustomDate1= fileType= cs1= cs1Label=MailRecipient suser= cs5= cs5Label=MailboxAccessType cnt= cs6= cs6Label=ChangedPermissions oldFilePermission= filePermission=Write dpriv=contosofoo.com\Group Admins-Region_200 start=Apr 28 2025 14:53:03

‎packages/varonis/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
- version: "0.2.0"
2+
changes:
3+
- description: Add preprocessors config option, to allow running processors before CEF messages are decoded.
4+
type: enhancement
5+
link: https://github.com/elastic/integrations/pull/13822
16
- version: "0.1.0"
27
changes:
38
- description: Initial release.

‎packages/varonis/data_stream/logs/_dev/test/system/test-tcp-invalid-cef-config.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
service: varonis-tcp-invalid-cef
2+
service_notify_signal: SIGHUP
3+
input: tcp
4+
data_stream:
5+
vars:
6+
tcp_host: 0.0.0.0
7+
tcp_port: 9033
8+
preserve_original_event: true
9+
tags:
10+
- forwarded
11+
preprocessors: |
12+
- script:
13+
lang: javascript
14+
description: Escape backslash characters.
15+
source: >
16+
function process(event) {
17+
var m = event.Get("message");
18+
event.Put("message", m.replace(/\\/g, "\\\\"));
19+
return event;
20+
}
21+
function test() {
22+
if ("\\".replace(/\\/g, "\\\\") !== "\\\\") {
23+
throw "expected replace \\ to \\\\ failed";
24+
}
25+
}
26+
assert:
27+
- hit_count: 2

‎packages/varonis/data_stream/logs/_dev/test/system/test-udp-invalid-cef-config.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
service: varonis-udp-invalid-cef
2+
service_notify_signal: SIGHUP
3+
input: udp
4+
data_stream:
5+
vars:
6+
udp_host: 0.0.0.0
7+
udp_port: 9032
8+
preserve_original_event: true
9+
tags:
10+
- forwarded
11+
preprocessors: |
12+
- script:
13+
lang: javascript
14+
description: Escape backslash characters.
15+
source: >
16+
function process(event) {
17+
var m = event.Get("message");
18+
event.Put("message", m.replace(/\\/g, "\\\\"));
19+
return event;
20+
}
21+
function test() {
22+
if ("\\".replace(/\\/g, "\\\\") !== "\\\\") {
23+
throw "expected replace \\ to \\\\ failed";
24+
}
25+
}
26+
assert:
27+
- hit_count: 2

‎packages/varonis/data_stream/logs/agent/stream/tcp.yml.hbs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,27 @@ publisher_pipeline.disable_host: true
1717
{{/contains}}
1818

1919
processors:
20+
{{#if preprocessors}}
21+
- copy_fields:
22+
fields:
23+
- from: "message"
24+
to: "@metadata.event_original"
25+
{{preprocessors}}
26+
{{/if}}
2027
- rename:
2128
fields:
2229
- {from: "message", to: "event.original"}
2330

2431
- decode_cef:
2532
field: event.original
2633

34+
{{#if preprocessors}}
35+
- convert:
36+
mode: rename
37+
fields:
38+
- from: "@metadata.event_original"
39+
to: "event.original"
40+
{{/if}}
2741
{{#if processors}}
2842
{{processors}}
2943
{{/if}}

‎packages/varonis/data_stream/logs/agent/stream/udp.yml.hbs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,27 @@ publisher_pipeline.disable_host: true
1313
{{/contains}}
1414

1515
processors:
16+
{{#if preprocessors}}
17+
- copy_fields:
18+
fields:
19+
- from: "message"
20+
to: "@metadata.event_original"
21+
{{preprocessors}}
22+
{{/if}}
1623
- rename:
1724
fields:
1825
- {from: "message", to: "event.original"}
1926

2027
- decode_cef:
2128
field: event.original
2229

30+
{{#if preprocessors}}
31+
- convert:
32+
mode: rename
33+
fields:
34+
- from: "@metadata.event_original"
35+
to: "event.original"
36+
{{/if}}
2337
{{#if processors}}
2438
{{processors}}
2539
{{/if}}

‎packages/varonis/data_stream/logs/fields/fields.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@
3535
- name: device_event_category
3636
type: keyword
3737
description: The category of the device event.
38+
- name: device_host_name
39+
type: keyword
40+
description: The host name of the device.
3841
- name: device_receipt_time
3942
description: The time the device received the event.
4043
- name: base_event_count

‎packages/varonis/data_stream/logs/manifest.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,15 @@ streams:
3939
default:
4040
- cef
4141
- forwarded
42+
- name: preprocessors
43+
title: Pre-Processors
44+
type: yaml
45+
description: >
46+
Pre-processors are run before the CEF message is decoded. They can be used to correct CEF formatting inconsistencies that may exist from some sources.
47+
See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
48+
required: false
49+
show_user: false
50+
multi: false
4251
- name: processors
4352
type: yaml
4453
title: Processors
@@ -86,6 +95,15 @@ streams:
8695
default:
8796
- cef
8897
- forwarded
98+
- name: preprocessors
99+
title: Pre-Processors
100+
type: yaml
101+
description: >
102+
Pre-processors are run before the CEF message is decoded. They can be used to correct CEF formatting inconsistencies that may exist from some sources.
103+
See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details.
104+
required: false
105+
show_user: false
106+
multi: false
89107
- name: processors
90108
type: yaml
91109
title: Processors

0 commit comments

Comments
 (0)