Elasticsearch not ingesting logs Redis queue not draining even with Logstash stopped.

[Bal33p]

Version

2.4.30

Installation Method

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

Description

other (please provide detail below)

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

63G

Storage for /

500

Storage for /nsm

30tb

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I tried resetting the Elasticsearch password, but it did not work. I have looked at the firewall, and I am not seeing anything obvious.

Can anyone tell if this is firewall-related or Elasticsearch authentication?

logstash_error.log
securityonion.log
redis-server.log

 [2023-12-27T14:45:41,942][INFO ][logstash.outputs.elasticsearch] Failed to perform request {:message=>"Connect to orseconip01:9200 [orseconip01/172.24.3.44] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to orseconip01:9200 [orseconip01/172.24.3.44] failed: Connection refused>}

[2023-12-27T14:45:42,036][WARN ][logstash.inputs.redis ] Redis connection error {:message=>"Error connecting to Redis on orseconrp01:9696 (Redis::TimeoutError)", :exception=>Redis::CannotConnectError}

it definitely seems like a firewall issue. I can't figure out how to fix it

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[dougburks]

Network installation on Red Hat derivative like Oracle, Rocky, Alma, etc.

You've started many discussions recently and it seems like you've made a lot of changes to your deployment. It might be the best use of your time to start over using our Security Onion ISO image:
https://docs.securityonion.net/en/2.4/os.html#supported

Make sure you are following best practices:
https://docs.securityonion.net/en/2.4/best-practices.html

We've discussed architecture previously, so it might be a good opportunity to make sure you are building the correct architecture:
https://docs.securityonion.net/en/2.4/architecture.html

[Bal33p]

My main question is, how can I keep control of my already deployed elastic agents if I reinstall over top of the existing server?
Won't the new install generate new keys, and then I will lose control?
Any guidance on how to keep the fleet or reattach the fleet would be appreciated.

[Bal33p]

Also for the record i was able to go into the Logstash docker container and connect to Elasticsearch
So it is not firewall related and the so_elastic and so_logstash username and passwords do work