Elastalert and Elasticfleet Not Starting Causing "Kibana server is not ready yet."

[Nicalicious]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

32GB

Storage for /

500gb (3 drives RAID0)

Storage for /nsm

500GB (default)

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

This is a brand new fresh install using the iso of Security Onion 2.4.8 on an esxi host 6.5 E5620 @ 2.40GHz. Used default settings. For esxi install. Two NICS. One for promiscuous mode and one for management.

Updated the SSL certificate using the SOC GUI to use my internal CA.

Other than that everything is default installation.

Oddly enough there are 0 logs or files in /opt/so/log/elastalert/ or in /opt/so/log/elasticfleet/.

Attempted a "sudo salt-call mine.update" and a "sudo so-checkin" before a "sudo soup -y". Then tried to manually start "sudo so-start elastalert" and "sudo so-start elasticfleet". Still does not start. Also tried to refresh the docker images with "so-docker-refresh"

Since there are no logs of the services it is hard to pinpoint the root of the issue.
When tailing /opt/so/log/elasticsearch/securityonion.log I get

[root@kibana log]# tail elasticsearch/securityonion.log
[2024-07-14T18:41:13,389][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-auditor]
[2024-07-14T18:57:17,996][WARN ][org.elasticsearch.http.netty4.Netty4HttpServerTransport] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.17.1.22:9200, remoteAddress=/172.17.1.1:32816}
[2024-07-14T19:11:07,072][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [analyst]
[2024-07-14T19:11:07,224][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [auditor]
[2024-07-14T19:11:07,375][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-analyst]
[2024-07-14T19:11:07,545][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-auditor]
[2024-07-14T19:58:16,602][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [analyst]
[2024-07-14T19:58:16,747][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [auditor]
[2024-07-14T19:58:16,892][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-analyst]
[2024-07-14T19:58:17,031][INFO ][org.elasticsearch.xpack.security.action.role.TransportPutRoleAction] updated role [limited-auditor]

sudo salt-call state.highstate provides similar errors to the /root/sosetup.log
{'pid': 262630, 'retcode': 0, 'stdout': 'Waiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings\' (1/300)\nServer is not ready\nWaiting for value 'fleet' at 'http://localhost:5601/api/fleet/settings\'

There are errors during the installation. Doing a tail of /root/sosetup.log I get (299/300)\nServer is not ready\nWaiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (300/300)\nServer is not ready\nServer still not ready after 300 attempts; giving up.", 'stderr': ''}

Tried reinstalling fresh several times. Same result every time.

I'm at a loss of what steps to take. As this is a fresh install that fails every time the only thing I can think of it is one of two things.

1. The docker images do not pull and install properly for some reason.
2. The licensing for elasticsearch/elastalert is expired/invalid and the service won't start.
   This is the output for checking the license. with sudo so-elasticsearch-query _license

{
"license" : {
"status" : "active",
"uid" : "2c08b447-fb31-4759-b3d0-493b00ec3908",
"type" : "basic",
"issue_date" : "2024-07-13T21:24:33.952Z",
"issue_date_in_millis" : 1720905873952,
"max_nodes" : 1000,
"max_resource_units" : null,
"issued_to" : "securityonion",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}

referenced this issue as a troubleshooting step as well #12648

Any guidance would be greatly appreciated.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What does this return? sudo so-elasticsearch-query _cat/shards | grep kibana