Playbook alerts not following custom filters

[DrOnionMaster]

Hey all,

I am running into an issue where the custom filters I build in the playbook are not working correctly. Here's an example:

I am trying to get rid of alerts for machine accounts that end with $
The filter looks like this:
sofilter:
event_data.user.name:
- *$
*The second line is once indented and the third is twice indented

As far as I understand this follows YAML formatting and should work.
I can confirm that after I put this filter into the custom filters and submit it, the filter shows up in the ElastAlert Config:
any where ( not ((event_data.user.name : "*$" ...

I checked the file in /opt/so/rules/elastalert/playbook/ and it also appears to be correct:

filter:
- eql: >
any where ( not ((event_data.user.name : "*$" ...

I've run sudo so-playbook-sync even though I know the file already looks correct.

But I am still continuing to get alerts for events where the event_data.user.name ends with $

Thanks for any help in advance.

[stormdosertao]

Hi DrOnionMaster,

are you already try to run so-elastialert-test to see one output more verbose about when this rule run?

[DrOnionMaster]

So I tried it for the alert I'm having problems with and this is what I got:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/test_rule.py", line 511, in main
    test_instance.run_rule_test()
  File "/usr/local/lib/python3.11/site-packages/elastalert/test_rule.py", line 457, in run_rule_test
    rule_yaml = conf['rules_loader'].load_yaml(self.args.file)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 253, in load_yaml
    loaded = self.get_yaml(current_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/loaders.py", line 585, in get_yaml
    return read_yaml(filename)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elastalert/yaml.py", line 6, in read_yaml
    with open(path) as f:
         ^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/elastalert/rules/playbook/1540ec71c.yaml'

One thing I noticed is that it appears the file path it is referencing is wrong. As you can see above its looking for the play in /opt/elastalert/rules/playbook but I don't even have that directory. The only path I can find is /opt/so/rules/elastalert/playbook/ is this an error in the way so-elastialert-test is built or am I doing something totally wrong here?

[DrOnionMaster]

The problem I am describing above seems to be another issue where the yaml file was deleted. I am not sure how this happened but I will create another issue for that because this problem is different than the one described in this thread.

The issue I need help with is still that the custom filters are not being followed. I tested this on another alert that still exists. I am using the same custom filter:

sofilter:
 event_data.user.name:
  - *$

I tried running so-elastalert-test and it was successful, but I don't think the output is super helpful in diagnosing the problem. Here's the output I get:

Got N/A hits from the last 0 day

Available terms in first hit:
        metadata.input.beats.host.ip
        metadata.raw_index
        metadata.stream_id
        ...
        user.domain
        user.name
        user.id

*Note I deleted most of the "Available Terms" to reduce the size of this comment. Hence the "..."

I'm not sure why it says "N/A hits from the last 0 day"? Do I need to specify the time range to test? If so, where do I do this? Because I have alerts within the Security Onion Console for this play so I know that it should be getting hits.

[dougburks]

Please try the new Detections interface in 2.4.70:
https://docs.securityonion.net/en/2.4/detections.html