so-elastalert missing container

[attinaibhayo]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

18

RAM

64gb

Storage for /

314.4 GB

Storage for /nsm

12263.4 GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Security Onion version: 2.4.
Deployment : on-prem with Internet access
Install Security Onion : Installed from SO ISO image
How many nodes do you have? 1

While checking service status we seen "so-elastalert missing"
Service satus : so-elastalert missing.
Security Onion Status
Container │ Status │ Details
───────────────────────────────────┼─────────┼──────────────────────
so-dockerregistry │ running │ Up 2 hours
so-elastalert │ missing │
so-elastic-fleet │ running │ Up 2 hours
so-elastic-fleet-package-registry │ running │ Up 2 hours (healthy)
so-idstools │ running │ Up 2 hours
so-influxdb │ running │ Up 2 hours (healthy)
so-kibana │ running │ Up 2 hours
so-kratos │ running │ Up 2 hours
so-logstash │ running │ Up 2 hours
so-mysql │ running │ Up 2 hours (healthy)
so-nginx │ running │ Up 2 hours (healthy)
so-playbook │ running │ Up 2 hours
so-redis │ running │ Up 2 hours
so-sensoroni │ running │ Up 2 hours
so-soc │ running │ Up 2 hours
so-soctopus │ running │ Up 2 hours
so-steno │ running │ Up 2 hours
so-strelka-backend │ running │ Up 2 hours
so-strelka-coordinator │ running │ Up 2 hours
so-strelka-filestream │ running │ Up 2 hours
so-strelka-frontend │ running │ Up 2 hours
so-strelka-gatekeeper │ running │ Up 2 hours
so-strelka-manager │ running │ Up 2 hours
so-suricata │ running │ Up 2 hours
so-telegraf │ running │ Up 2 hours
so-zeek │ running │ Up 2 hours (healthy)

I ran Command " sudo salt-call state.highstate" and got below output.
Output:

[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
local:
Data failed to compile:

The function "state.highstate" is running as PID 434342 and was started at 2024, Mar 13 06:42:42.410954 with jid 20240313064242410954

While checking for elastalert log file , it was missing.Below is the command i ran.
command: tail -f /opt/so/logs/elastalert/elastalert.log
Output:
tail: cannot open '/opt/so/logs/elastalert/elastalert.log' for reading: No such file or directory
tail: no files remaining

While i also search for elastichsearch using this " less /opt/so/log/elasticsearch/securityonion.log | grep "error"" i got the below result:
[2024-03-12T06:18:18,779][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-iis.error@custom]
[2024-03-12T06:18:18,786][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-iis.error@package]
[2024-03-12T06:18:19,567][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [logs-iis.error] for index patterns [logs-iis.error-]
[2024-03-12T06:18:55,567][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-mysql.error@custom]
[2024-03-12T06:18:55,571][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-mysql.error@package]
[2024-03-12T06:18:56,444][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [logs-mysql.error] for index patterns [logs-mysql.error-]
[2024-03-12T06:19:03,716][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-nginx.error@custom]
[2024-03-12T06:19:03,721][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [logs-nginx.error@package]
[2024-03-12T06:19:04,638][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [logs-nginx.error] for index patterns [logs-nginx.error-]
[2024-03-12T06:21:35,636][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding component template [error-mappings]
[2024-03-12T06:22:27,636][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [so-logs-apache.error] for index patterns [logs-apache.error-]
[2024-03-12T06:23:27,214][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [so-logs-iis.error] for index patterns [logs-iis.error-]
[2024-03-12T06:23:41,914][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [so-logs-mysql.error] for index patterns [logs-mysql.error-]
[2024-03-12T06:23:44,100][INFO ][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService] adding index template [so-logs-nginx.error] for index patterns [logs-nginx.error-*]
[2024-03-12T06:31:14,894][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_united-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.fleet-agents-7][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/10]
[2024-03-12T08:25:27,149][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-90d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:27,296][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:27,556][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:28,240][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-90d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:28,422][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:28,616][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:28,976][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-7d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:29,172][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:29,362][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_united-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/10]
[2024-03-12T08:25:30,195][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T08:25:30,715][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-7d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:39,452][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:39,601][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-7d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:39,877][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_current-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.ds-metrics-endpoint.metadata-default-2024.03.12-000001][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/10]
[2024-03-12T10:12:40,118][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-90d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:40,291][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:40,558][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:40,743][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-30d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:41,176][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-monthly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:41,832][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-occurrences-7d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:42,303][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-weekly-aligned] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]
[2024-03-12T10:12:42,615][WARN ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [endpoint.metadata_united-default-8.10.2] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/10]
[2024-03-12T10:12:42,851][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [slo-summary-timeslices-90d-rolling] Transform encountered an exception: [Failed to execute phase [query], ; org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.slo-observability.sli-v2][0]]. Consider using allow_partial_search_results setting to bypass this error.]; Will automatically retry [1/-1]

Can any one suggest what is happening here ?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[jaypfe]

Try adjusting your Heap config.

Also if you have any custom rules, disable them.

Giving a few more GB's to ES HEAP brought mine back

[attinaibhayo]

Thank you. I provided 23 GB to HEAP but did not solved my problem.

[attinaibhayo]

I now have done fresh installation but with different issue i.e data receiving on promicious interface not shown in kibana dashboard.

[capitangiaco]

In my case so-elastalert stopped when the disk usage reached the 85% of the /nsm partition on the search/master nodes.
85% is the default for the low watermark in elasticsearch, and it stops to allocate new shards.