SO 2.4.70 Suricata Rule Mismatch

[geistchevalier]

Version

2.4.70

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

64GB

Storage for /

500GB

Storage for /nsm

2TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi, I just upgraded to 2.4.70 from 2.4.60 and I am encountering the Suricata: Rule Mismatch warning

aside from that sudo so-status returns all green and sudo salt-call state.highstate does not get any failures

Summary for local
--------------
Succeeded: 829 (changed=35)
Failed:      0
--------------
Total states run:     829
Total run time:    72.169 s

Looking at the docs https://docs.securityonion.net/en/latest/detections.html#rule-engine-status, I checked if I had any custom rule in the following files:

/opt/so/saltstack/local/salt/idstools/rules/local.rules
/opt/so/saltstack/local/salt/suricata/rules/local.rules

but the files are empty

I checked the log at /opt/so/log/soc/detections_runtime-status_sigma.log and found a lot errors like the following that are either unknown or syntax error

{"_timestamp":"2024-05-31T00:08:17.138087Z","rule.name":"Potential PsExec Remote Execution - 3c28ba104","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'parsing_exception', 'line 1:107: token recognition error at: \\'\"* \\\\\\\\\\\\\\\\\\\\*\\'')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Syntax Error"}
{"_timestamp":"2024-05-31T00:08:18.524723Z","rule.name":"Potential RemoteFXvGPUDisablement.EXE Abuse - PowerShell ScriptBlock - fa7773519","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'planning_exception', 'Found 4 problems\\nline 1:36: Unresolved expression\\nline 1:36: Unresolved expression\\nline 1:36: Unresolved expression\\nline 1:36: Unresolved expression')","detection_type":"sig>
{"_timestamp":"2024-05-31T00:08:18.803775Z","rule.name":"Potential RemoteFXvGPUDisablement.EXE Abuse - PowerShell Module - ddfe9c955","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\\nline 1:131: Unknown column [winlog.event_data.Payload]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2024-05-31T00:08:18.924058Z","rule.name":"Potential Ursnif Malware Activity - Registry - 4ee5f679e","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\\nline 1:354: Unknown column [winlog.event_data.EventType]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2024-05-31T00:08:18.979953Z","rule.name":"Credential Dumping Attempt Via WerFault - 807c96d8e","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\\nline 1:55: Unknown column [winlog.event_data.GrantedAccess]\\nline 1:104: Unknown column [winlog.event_data.TargetImage]')","detection_type":"sigma","event_module":"soc","event_dataset":">
{"_timestamp":"2024-05-31T00:08:19.014258Z","rule.name":"Suspicious Cobalt Strike DNS Beaconing - DNS Client - cffdc214a","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\\nline 1:130: Unknown column [winlog_channel]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2024-05-31T00:08:19.391333Z","rule.name":"CobaltStrike Named Pipe Pattern Regex -- 0e7163d4-9e19-4fa7-9be6-000c61aad77a","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \\'\"\\\\\\\\mojo\\\\.\\'')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Syntax Error"}
{"_timestamp":"2024-05-31T00:08:19.483207Z","rule.name":"MSSQL Server Failed Logon From External Network - ec52fee92","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\\nline 1:19: Unknown column [Data]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2024-05-31T00:08:19.501873Z","rule.name":"Potential Privilege Escalation To LOCAL SYSTEM - c423f9b1b","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'x_content_parse_exception', '[eql] query malformed, no start_object after query name')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}

Is there anyway for me to remedy this situation?
Is there any other location I can check to delete custom rules for suricata?
Is there a way to reset and disable all the suricata rules so I can start fresh?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Koen1999]

[dougburks]

@Koen1999 Please review the discussion guidelines at #1720:

Start a new discussion instead of replying to somebody else's discussion
Please search to see if you can find similar discussions that may help you. However, in order to avoid confusion, please do NOT reply to somebody else's discussion with your own issue. Instead, please start a new discussion and in that new discussion you can provide a hyperlink to the related discussion.

[dougburks]

@geistchevalier Have you tried doing a FULL UPDATE to see if that helps?
https://docs.securityonion.net/en/2.4/detections.html#options

[geistchevalier]

Running full update only changes the status to pending and then back to rule mismatch when it finishes

Are there any other logs that I should look at in SO? I can't check it right now, but I'll check it as soon as possible and share my findings

[defensivedepth]

Click on the rule mismatch text and that will bring you to a set of logs associated with the syncing process. Filter for the event.action: integrity check report - drill down into the logs and post the message field here. It should look like something like this:

{"fields":{"deployedButNotEnabled":[],"detectionEngine":"suricata","enabledButNotDeployed":[],"intCheckId":"cfb85712-20e4-4d3e-9b42-21d125eacc78"},"level":"info","timestamp":"2024-05-31T11:37:17.658007898Z","message":"integrity check report"}

[geistchevalier]

It starts with a log with an event.action of suricata community diff

{"fields":{"syncAdded":0,"syncDeleteUnreferenced":true,"syncErrors":{},"syncRemoved":0,"syncUnchanged":49316,"syncUpdated":0},"level":"info","timestamp":"2024-06-03T22:41:30.840719184Z","message":"suricata community diff"}

and then suricata community rules sync finished

{"fields":{"durationSeconds":13.332364256},"level":"info","timestamp":"2024-06-03T22:41:34.636034794Z","message":"suricata community rules sync finished"}

and then waiting for next suricata community rules sync

{"fields":{"expectedStartTime":"2024-06-03T22:46:34Z","forceSync":true,"lastSyncSuccess":"false","waitTimeSeconds":300},"level":"info","timestamp":"2024-06-03T22:41:34.636042082Z","message":"waiting for next suricata community rules sync"}

and finally integrity check failed

{"fields":{"detectionEngine":"suricata","intCheckId":"f8ed7221-624d-47f2-9ad5-84a4d7d0eb78"},"level":"info","timestamp":"2024-06-03T22:41:34.636025925Z","message":"integrity check failed"}

[defensivedepth]

@geistchevalier Do you have any failures when running sudo salt-call state.highstate ?

[defensivedepth]

Really need to see the event.action: integrity check report log. see here: #13109 (reply in thread)

[geistchevalier]

These are the first three:

{"fields":{"deployedButNotEnabled":[],"detectionEngine":"suricata","enabledButNotDeployed":[],"intCheckId":"809f46f8-b7ba-4367-9757-8fe7c1691e6f"},"level":"info","timestamp":"2024-05-31T00:18:15.885547271Z","message":"integrity check report"}

{"fields":{"deployedButNotEnabled":["2053156","2053127","2053053","2053108","2053056","2522766","2053076","2053045","2053155","2053189","2053042","2053100","2053038","2053041","2053054","2053146","2053118","2053078","2053097","2053131","2053083","2053153","2053057","2053166","2053081","2053044","2053070","2053101","2053145","2053099","2053050","2053158","2053140","2053182","2053150","2053102","2053116","2053194","2053090","2053088","2053129","2053139","2053032","2053176","2053137","2053113","2053190","2053152","2053072","2053095","2053103","2053077","2053087","2053154","2053181","2053084","2053134","2053136","2053092","2053033","2053193","2053096","2053046","2053126","2053162","2053115","2053175","2053051","2053098","2053173","2053064","2053125","2053170","2053031","2053197","2053143","2053130","2053059","2053071","2053106","2053114","2053067","2053075","2522768","2053104","2053159","2053122","2053085","2053199","2053069","2053036","2053117","2053066","2053124","2053107","2053144","2053089","2053121","2053040","2053037","2053164","2053068","2053172","2053180","2053187","2053141","2053086","2053110","2053184","2053062","2053192","2053171","2053091","2053186","2522770","2053049","2053120","2053191","2053198","2053147","2053174","2522767","2053185","2053132","2053128","2053080","2053112","2053061","2053188","2053183","2053055","2053082","2053165","2053111","2525008","2053047","2053048","2522769","2053149","2053167","2053123","2053179","2053195","2053168","2053135","2053065","2053060","2053039","2053035","2053093","2053138","2053043","2053105","2053160","2053052","2053196","2053119","2053109","2053074","2053142","2053161","2053178","2053079","2053063","2053094","2053151","2053148","2053073","2053030","2053163","2053177","2525007","2053034","2053157","2053058","2053169","2053133"],"detectionEngine":"suricata","enabledButNotDeployed":["2404303","2520110"],"intCheckId":"2ff05059-1424-4498-a333-eb920a0aaf1d"},"level":"info","timestamp":"2024-05-31T00:28:17.618770809Z","message":"integrity check report"}

{"fields":{"deployedButNotEnabled":[],"detectionEngine":"suricata","enabledButNotDeployed":["2520110"],"intCheckId":"d23328db-b603-41b4-bbdb-30da0a9f1726"},"level":"info","timestamp":"2024-05-31T00:29:17.383200805Z","message":"integrity check report"}

There's a 13 more but it's the same "message":"integrity check report" with different GUID/UUID for the check id and empty "enabledButNotDeployed":[], probably from the times I pressed Full Update

---

There are other event.actions after I updated SO

event.action: suricata has successfully migrated to 2.4.70
message: {"fields":{"errMap":null},"level":"info","timestamp":"2024-05-31T00:15:59.934658979Z","message":"suricata has successfully migrated to 2.4.70"}

event.action: Updating custom file setting to new value
message: {"fields":{"settingId":"suricata.thresholding.sids__yaml","settingLength":3,"settingPath":"/opt/so/saltstack/local/salt/suricata/thresholding/sids.yaml"},"level":"info","timestamp":"2024-05-31T00:15:58.394930644Z","message":"Updating custom file setting to new value"}

with .../thresholding/sids.yaml containing just

{}

event.action: suricata community diff
first message after the update:
{"fields":{"syncAdded":49140,"syncDeleteUnreferenced":true,"syncErrors":{"2043788":"unable to create detection; reason=ERROR_QUERY_FAILED_ELASTICSEARCH"},"syncRemoved":0,"syncUnchanged":0,"syncUpdated":0},"level":"info","timestamp":"2024-05-31T00:15:58.394996874Z","message":"suricata community diff"}
vs the latest one:
{"fields":{"syncAdded":0,"syncDeleteUnreferenced":true,"syncErrors":{},"syncRemoved":0,"syncUnchanged":49490,"syncUpdated":0},"level":"info","timestamp":"2024-06-10T03:37:31.07459374Z","message":"suricata community diff"}

[geistchevalier]

@defensivedepth additional details,

There's not supposed to be anything from playbook after the upgrade to 2.4.70 correct?
but I can still trigger detections that are apart of the playbook event.module

looking at #13127

I tried looking at /opt/so/rules/elastalert/playbook and got

Running sudo ls /nsm/backup/detections-migration/sigma/rules returns nothing, meaning an empty folder

Should I manually move the files in /opt/so/rules/elastalert/playbook to /nsm/backup/detections-migration/sigma/rules like the solution in that discussion?

[defensivedepth]

Did you see this when you ran soup?

[geistchevalier]

I can't recall if I saw Active Elastalert/Playbook rules found.
Here's what I'm seeing in my soup.log when I search for playbook

Found salt-master PID 1074

Stopping salt-master service at 22:56:27.881958
Successfully stopped salt-master.

Checking to see if changes are needed.

Stopping Playbook services & cleaning up...
so-playbook
so-mysql
so-soctopus

Playbook Migration is complete....
sending incremental file list
adv_idstools.sls
soc_idstools.sls

sent 185 bytes  received 54 bytes  478.00 bytes/sec
total size is 0  speedup is 0.00
IDStools configuration has been backed up.
sending incremental file list
thresholding/

sent 84 bytes  received 20 bytes  208.00 bytes/sec
total size is 0  speedup is 0.00
Suricata thresholds have been backed up.
sending incremental file list
local.rules

sent 160 bytes  received 35 bytes  390.00 bytes/sec
total size is 46  speedup is 0.24

5962:Stopping Playbook services & cleaning up...
5967:Playbook Migration is complete....
15459:                      +exclude_container so-playbook # Playbook is removed as of 2.4.70, disregard output in stopped containers
15474:                      +exclude_log "playbook.log" # Playbook is removed as of 2.4.70, logs may still be on disk

And then there's a bunch of diff log

-exclude_container so-playbook # ignore due to several playbook known issues
+exclude_container so-playbook # Playbook is removed as of 2.4.70, disregard output in stopped containers

Result at the end of 2.4.70 soup

Summary for local
--------------
Succeeded: 829 (changed=92)
Failed:      0
--------------
Total states run:     829
Total run time:   632.406 s
Running post upgrade processes.

Removing idh.services from any existing IDH node pillar files

Upgrade to 2.4.70 complete.
Checking the number of minions.
Checking sudoers file.
There is an entry for so-setup in the sudoers file, this can be safely deleted using "visudo".

Starting crond service at 23:17:45.011415
Successfully started crond.

### soup has been served at Thu May 30 11:17:45 PM UTC 2024 ###

[defensivedepth]

@geistchevalier 2.4.80 was released yesterday and has a number of fixes for Detections. I would suggest upgrading and seeing if that fixes the issue.

[geistchevalier]

Seems like it's fixed, I'll observe the status for a few more days and see if any other issues arise