Upgrade to 2.4.190, Suricata: Rule Mismatch

[Green-Pool]

Version

2.4.190

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

48 (2 sockets, 24 cores) [x86-64-v2-AES]

RAM

150GB [balloon=0]

Storage for /

300G

Storage for /nsm

3000G

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Standalone installation used domestically, partially for self learning and partially as the household's SIEM. Installation is about a year old and has successfully upgraded a number of times in the past. This time around, I upgraded from 2.4.170 (I think, but wasn't really paying attention) to 2.4.190. Post upgrade, I also enabled ti_abusech in the Windows and Linux endpoint elastic tools with my API key. All seemed to work OK at the time and I didn't pay much further attention. But a few days later I noticed that previously suppressed suricata alerts were now appearing and that the "tune detection" in the drop down list is greyed out. An exclamation mark has appeared in the Detections page.

ElastAlert: OK
Strelka: OK
Suricata: Rule Mismatch

I have followed the prior discussion on #13109 for additional guidance, with similar results.

sudo so-status returns all green

These rule files are empty

/opt/so/saltstack/local/salt/idstools/rules/local.rules
/opt/so/saltstack/local/salt/suricata/rules/local.rules 

The log file /opt/so/log/soc/detections_runtime-status_sigma.log has a lot of errors. Small extract below

{"_timestamp":"2025-11-03T05:39:21.630973Z","rule.name":"Silence.EDA Detection -- 3ceb2083-a27f-449a-be33-14ec1b7cc973","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 12 problems\\nline 1:241: [:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead\\nline 1:310: [:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead\\nline 1:366: [:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead\\nline 1:425: [:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead\\nline 1:490: [:] cannot operate on first argument field of data type [text]: No keyword/multi-field defined exact matches for [script_block_text]; define one or use MATCH/QUERY instead\\nline 1:541: [:] cannot operate on f... (1260 characters removed)","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2025-11-03T05:39:37.270695Z","rule.name":"Bitbucket Unauthorized Full Data Export Triggered -- 34d81081-03c9-4a7f-91c9-5e46af625cde","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\\nline 1:11: Unknown column [auditType.category]\\nline 1:50: Unknown column [auditType.action]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2025-11-03T05:39:44.393354Z","rule.name":"CobaltStrike Named Pipe Pattern Regex -- 0e7163d4-9e19-4fa7-9be6-000c61aad77a","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \\'\"\\\\\\\\mojo\\\\.\\'')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Syntax Error"}
{"_timestamp":"2025-11-03T05:39:45.462250Z","rule.name":"DiagTrackEoP Default Login Username -- 2111118f-7e46-4fc8-974a-59fd8ec95196","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\\nline 1:95: second argument of [:] must be [string], found value [9] type [integer]; consider using [==] instead')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}
{"_timestamp":"2025-11-03T05:39:49.346969Z","rule.name":"Possible Coin Miner CPU Priority Param -- 071d5e5a-9cef-47ec-bc4e-a42e34d8d0ed","error_type":"runtime_status","error_message":"Error running query: RequestError(400, 'verification_exception', 'Found 7 problems\\nline 1:11: Unknown column [a1]\\nline 1:35: Unknown column [a2]\\nline 1:59: Unknown column [a3]\\nline 1:83: Unknown column [a4]\\nline 1:107: Unknown column [a5]\\nline 1:131: Unknown column [a6]\\nline 1:155: Unknown column [a7]')","detection_type":"sigma","event_module":"soc","event_dataset":"soc.detections","error_analysis":"Unknown"}

There are logs in /opt/so/log/suricata/ but (to me) these don't seem to be related to the problem.

flow.recycler.queue_max                                      | Total                     | 38
tcp.memuse                                                   | Total                     | 10584144
tcp.reassembly_memuse                                        | Total                     | 3311272
http.memuse                                                  | Total                     | 2517
http.memcap                                                  | Total                     | 0
ftp.memuse                                                   | Total                     | 0
ftp.memcap                                                   | Total                     | 0
app_layer.expectations                                       | Total                     | 0
file_store.open_files                                        | Total                     | 0
flow.memuse                                                  | Total                     | 7509504

I have restarted Security Onion and also tried a SOUP, without any success.

I am presently out of ideas for possible next steps.

Thoughts?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

If you click on the Rule Mismatch in Detections it should pivot to Hunt. What are the error logs you see?

[Green-Pool]

Ahh, OK. I didn't realise that I could click on the Rule Mismatch to dig further.

This is the extract from the log

event.action | integrity check failed

{"fields":{"deployedButNotEnabled":["2042102","2026585","2062085","2004895","2005089","2005676","2006186","2042906","2064125","2061512","2012110","2016308","2010173","2060767","2028755","2043375","2023988","2032329","2036182","2006120"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"897694f2-5ae1-4d08-a8f9-fcde50218d58"},"level":"warn","timestamp":"2025-11-04T20:26:16.809837642Z","message":"integrity check failed"}

The log file path pointed to is /opt/so/log/soc/sensoroni-server.log The last few lines of which read:

{"fields":{"contentLength":0,"elapsedMs":578,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:57812","requestId":"52f32b26-9ccc-4e7a-93dc-56d3ea7dff57","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.60","statusCode":200},"level":"info","timestamp":"2025-11-04T20:42:37.983651928Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:57828","requestId":"c4f6a5ae-d66b-41ae-acd7-2cd1dc2a9c85","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.60","statusCode":200},"level":"info","timestamp":"2025-11-04T20:42:38.989001355Z","message":"Handled request"}

Grepping through the log for integrity check failed gives this result for the last 19 entry of the extract.

{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T17:26:02.939060946Z","message":"integrity check repeat failure, alerting user"}
{"fields":{"deployedButNotEnabled":["2043200","2019833","2040576","2043541","2046028","2062046","2043147","2062906","2030179","2058357","2051785","2049430","2061473","2045124","2056774","2041588","2038317","2006330","2006302","2010088"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"8e5caad3-4d44-4edb-858e-2a091b5b8d10"},"level":"warn","timestamp":"2025-11-04T17:46:05.459483558Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T17:46:05.459535379Z","message":"integrity check first failure, running force sync"}
{"fields":{"deployedButNotEnabled":["2016751","2046346","2020780","2038091","2039038","2051171","2015695","2032112","2021261","2045581","2017752","2004251","2061441","2029646","2058462","2025189","2023945","2018678","2059189","2054928"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"e96554a4-9649-4574-99ce-5e34b8a0c3d0"},"level":"warn","timestamp":"2025-11-04T18:46:08.377193736Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T18:46:08.37727231Z","message":"integrity check repeat failure, alerting user"}
{"fields":{"deployedButNotEnabled":["2038557","2051055","2522050","2048894","2029615","2016879","2025108","2058901","2031815","2040033","2014086","2033955","2042121","2049295","2065341","2024462","2522571","2058087","2046500","2023581"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"9a879020-ddfa-4e60-a757-0626d311b6b7"},"level":"warn","timestamp":"2025-11-04T19:06:11.006405725Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T19:06:11.006476105Z","message":"integrity check first failure, running force sync"}
{"fields":{"deployedButNotEnabled":["2033226","2056271","2021631","2046388","2006980","2033280","2050605","2044249","2046892","2033545","2040048","2007518","2008830","2020123","2033522","2060416","2062854","2018076","2060361","2051600"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"2ae3f70b-a79b-4778-ab09-f95c40e59f46"},"level":"warn","timestamp":"2025-11-04T20:06:14.107895483Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T20:06:14.107974952Z","message":"integrity check repeat failure, alerting user"}
{"fields":{"deployedButNotEnabled":["2042102","2026585","2062085","2004895","2005089","2005676","2006186","2042906","2064125","2061512","2012110","2016308","2010173","2060767","2028755","2043375","2023988","2032329","2036182","2006120"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"897694f2-5ae1-4d08-a8f9-fcde50218d58"},"level":"warn","timestamp":"2025-11-04T20:26:16.809837642Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T20:26:16.809883924Z","message":"integrity check first failure, running force sync"}
{"fields":{"contentLength":271,"elapsedMs":13,"impl":"/build/server/queryhandler.go:196:github.com/security-onion-solutions/securityonion-soc/server.(*QueryHandler).getQuery","remoteAddr":"172.17.1.1:50462","requestId":"6799919e-6651-49ce-b3e4-e434f182d4a7","requestMethod":"GET","requestPath":"/api/query/filtered","requestQuery":{"condense":["true"],"field":[""],"mode":["INCLUDE"],"query":["event.action: \"integrity check failed\" AND soc.fields.detectionEngine:\"suricata\" | table event.dataset soc.fields.deployedButNotEnabledCount soc.fields.enabledButNotDeployedCount soc.fields.syncId"],"scalar":["true"],"value":["NOT _index:\"*:so-case*\" AND NOT _index:\"*:so-assistant-*\""]},"requestorId":"0715f353-68ef-4fbc-b51b-2c2411439272","sourceIp":"192.168.7.5","statusCode":200},"level":"info","timestamp":"2025-11-04T20:36:25.25396106Z","message":"Handled request"}
{"fields":{"query":"{\"aggs\":{\"timeline\":{\"date_histogram\":{\"field\":\"@timestamp\",\"fixed_interval\":\"1m\",\"min_doc_count\":1}}},\"query\":{\"bool\":{\"filter\":[],\"must\":[{\"query_string\":{\"analyze_wildcard\":true,\"default_field\":\"*\",\"query\":\"(event.action: \\\"integrity check failed\\\" AND soc.fields.detectionEngine: \\\"suricata\\\") AND NOT _index: \\\"*:so-case*\\\" AND NOT _index: \\\"*:so-assistant-*\\\"\"}},{\"range\":{\"@timestamp\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2025-11-05T05:36:24+10:00\",\"lte\":\"2025-11-05T06:36:24+10:00\"}}}],\"must_not\":[],\"should\":[]}},\"size\":5000}","requestId":"6c423c89-1c33-4fd3-b380-66b1d33c9070"},"level":"info","timestamp":"2025-11-04T20:36:25.280917905Z","message":"Searching Elasticsearch"}
{"fields":{"contentLength":7585,"elapsedMs":542,"impl":"/build/server/eventhandler.go:93:github.com/security-onion-solutions/securityonion-soc/server.(*EventHandler).getEvent","remoteAddr":"172.17.1.1:50466","requestId":"6c423c89-1c33-4fd3-b380-66b1d33c9070","requestMethod":"GET","requestPath":"/api/events/","requestQuery":{"eventLimit":["5000"],"format":["2006/01/02 3:04:05 PM"],"metricLimit":["500"],"query":["(event.action: \"integrity check failed\" AND soc.fields.detectionEngine: \"suricata\") AND NOT _index:\"*:so-case*\" AND NOT _index:\"*:so-assistant-*\" | table event.dataset soc.fields.deployedButNotEnabledCount soc.fields.enabledButNotDeployedCount soc.fields.syncId"],"range":["2025/11/05 05:36:24 AM - 2025/11/05 06:36:24 AM"],"zone":["..."]},"requestorId":"0715f353-68ef-4fbc-b51b-2c2411439272","sourceIp":"192.168.7.5","statusCode":200},"level":"info","timestamp":"2025-11-04T20:36:25.811896533Z","message":"Handled request"}
{"fields":{"contentLength":271,"elapsedMs":11,"impl":"/build/server/queryhandler.go:196:github.com/security-onion-solutions/securityonion-soc/server.(*QueryHandler).getQuery","remoteAddr":"172.17.1.1:54578","requestId":"71652541-75db-49dc-bc40-c0f9129982b4","requestMethod":"GET","requestPath":"/api/query/filtered","requestQuery":{"condense":["true"],"field":[""],"mode":["INCLUDE"],"query":["event.action: \"integrity check failed\" AND soc.fields.detectionEngine:\"suricata\" | table event.dataset soc.fields.deployedButNotEnabledCount soc.fields.enabledButNotDeployedCount soc.fields.syncId"],"scalar":["true"],"value":["NOT _index:\"*:so-case*\" AND NOT _index:\"*:so-assistant-*\""]},"requestorId":"0715f353-68ef-4fbc-b51b-2c2411439272","sourceIp":"192.168.7.5","statusCode":200},"level":"info","timestamp":"2025-11-04T20:38:59.576527528Z","message":"Handled request"}
{"fields":{"query":"{\"aggs\":{\"timeline\":{\"date_histogram\":{\"field\":\"@timestamp\",\"fixed_interval\":\"1m\",\"min_doc_count\":1}}},\"query\":{\"bool\":{\"filter\":[],\"must\":[{\"query_string\":{\"analyze_wildcard\":true,\"default_field\":\"*\",\"query\":\"(event.action: \\\"integrity check failed\\\" AND soc.fields.detectionEngine: \\\"suricata\\\") AND NOT _index: \\\"*:so-case*\\\" AND NOT _index: \\\"*:so-assistant-*\\\"\"}},{\"range\":{\"@timestamp\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2025-11-05T05:38:54+10:00\",\"lte\":\"2025-11-05T06:38:54+10:00\"}}}],\"must_not\":[],\"should\":[]}},\"size\":5000}","requestId":"fe80d8e4-ece0-495d-981b-88215ef8d420"},"level":"info","timestamp":"2025-11-04T20:39:00.57984981Z","message":"Searching Elasticsearch"}
{"fields":{"contentLength":7584,"elapsedMs":1278,"impl":"/build/server/eventhandler.go:93:github.com/security-onion-solutions/securityonion-soc/server.(*EventHandler).getEvent","remoteAddr":"172.17.1.1:54590","requestId":"fe80d8e4-ece0-495d-981b-88215ef8d420","requestMethod":"GET","requestPath":"/api/events/","requestQuery":{"eventLimit":["5000"],"format":["2006/01/02 3:04:05 PM"],"metricLimit":["500"],"query":["(event.action: \"integrity check failed\" AND soc.fields.detectionEngine: \"suricata\") AND NOT _index:\"*:so-case*\" AND NOT _index:\"*:so-assistant-*\" | table event.dataset soc.fields.deployedButNotEnabledCount soc.fields.enabledButNotDeployedCount soc.fields.syncId"],"range":["2025/11/05 05:38:54 AM - 2025/11/05 06:38:54 AM"],"zone":["..."]},"requestorId":"0715f353-68ef-4fbc-b51b-2c2411439272","sourceIp":"192.168.7.5","statusCode":200},"level":"info","timestamp":"2025-11-04T20:39:00.879018496Z","message":"Handled request"}
{"fields":{"deployedButNotEnabled":["2042869","2032635","2036434","2044078","2006571","2522237","2061917","2046429","2064196","2055475","2032175","2032948","2062995","2054067","2004857","2034522","2052861","2023868","2006123","2061179"],"deployedButNotEnabledCount":45874,"detectionEngine":"suricata","enabledButNotDeployed":[],"enabledButNotDeployedCount":0,"intCheckId":"b8be51f2-e638-4bbb-a4cb-866703cbeeb6"},"level":"warn","timestamp":"2025-11-04T21:06:19.563644951Z","message":"integrity check failed"}
{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2025-11-04T21:06:19.563718764Z","message":"integrity check repeat failure, alerting user"}

Digging around a bit more, I followed this discussion #13112

I picked the first rule that is barfing 2042102, and following the help in 13112, the first step is to Is Enabled in Detections interface returns an answer of 95 sigma rules, 4321 yara rules, but no suricata rules.

Selecting the pre-canned rule so_detection.language:suricata | groupby so_detection.ruleset so_detection.isEnabled | groupby so_detection.category' returns No data available`

The contents of /opt/so/log/suricata/suricata.log are

[12 - Suricata-Main] 2025-11-04 07:14:16 Notice: detect: rule reload starting
[12 - Suricata-Main] 2025-11-04 07:14:20 Info: detect: 1 rule files processed. 45874 rules successfully loaded, 0 rules failed, 0
[12 - Suricata-Main] 2025-11-04 07:14:20 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[12 - Suricata-Main] 2025-11-04 07:14:21 Info: detect: 45877 signatures processed. 966 are IP-only rules, 4420 are inspecting packet payload, 40442 inspect application layer, 0 are decoder event only
[12 - Suricata-Main] 2025-11-04 07:14:25 Notice: detect: rule reload complete
[12 - Suricata-Main] 2025-11-04 07:14:26 Notice: detect: rule reload starting
[12 - Suricata-Main] 2025-11-04 07:14:30 Info: detect: 1 rule files processed. 45874 rules successfully loaded, 0 rules failed, 0
[12 - Suricata-Main] 2025-11-04 07:14:30 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[12 - Suricata-Main] 2025-11-04 07:14:30 Info: detect: 45877 signatures processed. 966 are IP-only rules, 4420 are inspecting packet payload, 40442 inspect application layer, 0 are decoder event only
[12 - Suricata-Main] 2025-11-04 07:14:35 Notice: detect: rule reload complete

I am stuck again as to how to proceed next.

[cm-ops]

Would you let me know what this shows, sudo so-index-list | grep so-detection please? It looks like your Suricata rules did not get imported into the index at first glance.

[Green-Pool]

Thanks for the help.

This is the result.

green  open   so-detection                                                           ZUzwMQo8T8CtuiZijhsGOg   1   0       6146        71572     49.3mb         49.3mb       49.3mb
green  open   so-detectionhistory                                                    rtTDXFanTluyFUD06v5P5Q   1   0     561168            0      341mb          341mb        341mb

[cm-ops]

It looks like your Suricata rules did not make it into the so-detection index.

Try this:
Remove /opt/so/conf/soc/fingerprints/emerging-all.fingerprint and /opt/so/conf/soc/fingerprints/suricataengine.state
Restart SOC - sudo so-soc-restart
Run a full sync on Suricata rules from Detections

[Green-Pool]

Thank you @cm-ops This has solved the problem.

The part of the output from the restart SOC said,

          ID: soc_sbin
    Function: file.recurse
        Name: /usr/sbin
      Result: True
     Comment: Recursively updated /usr/sbin
     Started: 21:17:21.607495
    Duration: 910.619 ms
     Changes:
              ----------
              /usr/sbin:
                  ----------
                  /usr/sbin:
                      ----------
                      user:
                          939
                  user:
                      939

Summary for local
-------------
Succeeded: 51 (changed=1)
Failed:     0
-------------
Total states run:     51
Total run time:   26.023 s

The error messages have disappeared, and I can tune detections again. Unfortunately, I have lost my customed tuning, but that is easy enough to rebuild (and much easier than rebuilding the full machine).

Thank you.