After upgrade to 2.4.50 I'm getting: Elasticsearch Status: Fault | so-elastalert missing

[jtchi717]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

16

Storage for /

1tb

Storage for /nsm

2tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Everything was working well with 2.4.30 until we decided to upgrade to 2.4.50. Right after the upgrade, it breaks the installation.
We backup the manager node only (Veeam) since search and sensor can get really large.
We tried to restart the manager node back to 2.4.3 but the problem is still there.
This is a simple deployment with 3 nodes like in text book/sample installation tutorial scenario.

Tried to restart the 3 nodes alternatively and full shutdown/restart.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[jtchi717]

like to add that it seems to "Stuck" at "Executing command 'so-elasticsearch-wait' in directory '/root' for a long time.
ends with this error:

---

      ID: so-elastalert
Function: docker_container.running
  Result: False
 Comment: One or more requisite failed: elastalert.enabled.wait_for_elasticsearch
 Started: 19:35:48.118465
Duration: 0.007 ms
 Changes:

---

      ID: delete_so-elastalert_so-status.disabled
Function: file.uncomment
    Name: /opt/so/conf/so-status/so-status.conf
  Result: True
 Comment: Pattern already uncommented
 Started: 19:35:48.118641
Duration: 4.697 ms
 Changes:

Summary for local

Succeeded: 12 (changed=2)
Failed: 2

[cm-ops]

Is Elasticsearch up? Are there any clues in the Elasticsearch log /opt/so/log/elasticsearch/securityonion.log?

[baulaa]

Hi,
I have same issue and here is the logs from /opt/so/log/elasticsearch/securityonion.log.
[2024-04-26T22:58:33,433][INFO ][org.elasticsearch.index.mapper.MapperService] reloading search analyzers
[2024-04-26T22:58:33,492][INFO ][org.elasticsearch.index.mapper.MapperService] reloading search analyzers
[2024-04-26T22:58:33,514][INFO ][org.elasticsearch.index.mapper.MapperService] reloading search analyzers
[2024-04-26T22:58:33,581][INFO ][org.elasticsearch.index.mapper.MapperService] reloading search analyzers
[2024-04-26T22:58:33,585][INFO ][org.elasticsearch.index.mapper.MapperService] reloading search analyzers
[2024-04-26T22:58:35,977][INFO ][org.elasticsearch.cluster.routing.allocation.AllocationService] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-logs-zeek-so-2024.04.26-000001][0]]])." previous.health="RED" reason="shards started [[.ds-logs-zeek-so-2024.04.26-000001][0]]"
[2024-04-26T22:58:36,755][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [analyst]
[2024-04-26T22:58:37,054][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [auditor]
[2024-04-26T22:58:37,221][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-analyst]
[2024-04-26T22:58:37,306][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-auditor]