SO 2.4.70 Update; Elastalert Missing, Kibana Inaccessable

[8inaryMata1eao]

I have 4 separate instances of SO at different client locations. I would say the update went as expected.

There is one instance that has a continual issue with no real indicator of the source. When accessing the grid dashboard in the manager detail shows "so-elastalert missing" on the right under "Container Status" and "Elasticsearch Status: Pending" on the left under node status. Even "so-status" shows everything green and working. When I ran so-elastalert-start, it would show waiting for elasticsearch and show 1/300 - 300/300 "Server is not ready". Yet the grid container status shows "so-elasticsearch running". After running "so-docker-refresh" a few times, many reboots and attempts at running "so-elastalert-start', I went into the configuration and set elastalert to false hoping that was the problem.

Another thing that seems odd is the elasticsearch log shows "[sch-01][XXXX:9300] Node not connected" but shows everything ok in the grid details for the search node.

I have not made any customization to Kibana and have done the most I can to troubleshoot this but due to the high coupling of Security Onion, one thing can break everything, I had to ask the pros for help.
Thanks

So-Status

Manger Status

SearchNode Status

Elasticsearch Log (grep error)
[root@so-mgr-01]# cat /opt/so/log/elasticsearch/securityonion.log | grep -i error

[2024-06-17T13:25:07,451][INFO ][org.elasticsearch.node.Node] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-7684465398313147706, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms5290m, -Xmx5290m, -Des.transport.cname_in_publish_address=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=2774532096, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=org.elasticsearch.preallocate, -Djdk.module.main=org.elasticsearch.server] [2024-06-17T13:25:31,870][ERROR][org.elasticsearch.ingest.geoip.GeoIpDownloader] exception during geoip databases update [2024-06-17T13:25:35,636][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,637][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,794][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,949][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:36,106][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_united-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/10] [2024-06-17T13:25:36,316][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:36,485][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1]

Elasticsearch Log (grep fail)
[root@so-mgr-01]# cat /opt/so/log/elasticsearch/securityonion.log | grep -i fail

[2024-06-17T13:20:36,813][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [osquery_manager.action_responses-default-0.0.1] Transform encountered an exception: [org.elasticsearch.transport.NodeNotConnectedException: [sch-01][XXXX:9300] Node not connected]; Will automatically retry [1/10] [2024-06-17T13:20:36,828][WARN ][org.elasticsearch.xpack.transform.transforms.ClientTransformIndexer] [osquery_manager.action_responses-default-0.0.1] updating stats of transform failed. java.lang.RuntimeException: Failed to persist transform statistics for transform [osquery_manager.action_responses-default-0.0.1] at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:171) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:625) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:171) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1026) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:777) ~[elasticsearch-8.10.4.jar:?] at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.10.4.jar:?] [2024-06-17T13:25:35,636][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,637][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,794][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:35,949][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:36,106][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_united-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/10] [2024-06-17T13:25:36,316][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1] [2024-06-17T13:25:36,485][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider usingallow_partial_search_resultssetting to bypass this error.]; Will automatically retry [1/-1]

Kibana Log (grep error)
[root@so-mgr-01]# grep -i -e fail /opt/so/log/kibana/kibana.log
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-06-17T13:27:30.458+00:00","message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","error":{"message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","type":"Error","stack_trace":"Error: Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations\n at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":6},"trace":{"id":"009a887604a1856ae6fc8e75254652ad"},"transaction":{"id":"07fdeec55dbcb850"}} {"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-06-17T15:11:11.800+00:00","message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","error":{"message":"Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations","type":"Error","stack_trace":"Error: Failure during installation of common resources shared between all indices. Server is stopping; must stop all async operations\n at installWithTimeout (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_service/lib/install_with_timeout.js:48:11)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at ResourceInstaller.installCommonResources (/usr/share/kibana/node_modules/@kbn/rule-registry-plugin/server/rule_data_plugin_service/resource_installer.js:42:5)"},"log":{"level":"ERROR","logger":"plugins.ruleRegistry"},"process":{"pid":6},"trace":{"id":"501bbeb6d848a2e236ce777a1782192c"},"transaction":{"id":"2036003da19954bd"}}

[cm-ops]

Check your shards, you most likely have shards that are not started due to your watermark threshold on the manager node.

Two commands to run: sudo so-elasticsearch-query _cat/shards | grep UNAS and ``sudo so-elasticsearch-query _cluster/allocation/explain pretty` If you are over the watermark threshold ES will not start and initialize shards on that node.

[8inaryMata1eao]

Thank you for the follow up. The NSM partition is at 77% and i have not modified the default.

sudo so-elasticsearch-query _cat/shards | grep UNAS

• Nothing Returned

sudo so-elasticsearch-query _cluster/allocation/explain pretty

[cm-ops]

I need to edit that allocation command, sudo so-elasticsearch-query _cluster/allocation/explain?pretty

[8inaryMata1eao]

Output:

Thanks again for the help.

[defensivedepth]

@8inaryMata1eao From: https://blog.securityonion.net/2024/05/security-onion-2470-now-available.html

Existing 2.4 Installations
If you have an existing Security Onion 2.4 installation, you can update to the latest version using soup:
https://docs.securityonion.net/en/2.4/soup.html

Before updating your production deployment, we highly recommend testing the upgrade process on a test deployment that closely matches your production deployment if possible.

In preparation for the new Detections module, the following will be completed during soup:

• Playbook Plays will be backed up to /nsm/backup/detections-migration/ and any active Elastalert rules will be backed up and removed.
• Suricata tuning configurations will be backed to /nsm/backup/detections-migration/ and any thresholds will be migrated over to the new Detections module.

---

Can you clarify what you mean when you say that your Suricata variables were removed?

[8inaryMata1eao]

Josh, that was my mistake. After going back and grepping through, it is all still there, I will remove that from the initial post. I am still having the issue with the elasticsearch. The backups have been critical to say the least.

[defensivedepth]

Ok understood. We are looking at reintroducing the regex enable|disable, but for new overrides you will need to bulk-select enable | disable in the Detections interface.

[8inaryMata1eao]

Awesome, I look forward to that returning. Thanks again.