After update no data is getting in the DB, we see data with a tcpdump, but nothing in db/dashboard. Guessing misconfigured firewall, but not sure.

[TheAOWizard]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

100

Storage for /nsm

500

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

We have updated our entire setup from 2.3 to 2.4.

Our setup consist of:
1 x Manager Node
5 x Search nodes
2 x Receiver nodes
3 x Forward nodes

All nodes are in the grid and all showing healthy.

Since the update we haven't received any data in our databases and nothing is showing up in our dashboard from after the update (we can go back and see data from before the update).

If we run a tcpdump on the manager node, we can see netflow data from our cisco enviroment.

Before the update, we got sysmon, netflow and data from a span port, everything was working great.
Now we get:

And in the influxDB:

But we are getting data on the monitor interface:

Our firewall config (not including default settings) looks like this:

firewall -> hostgroups:
analyst:
- 10.101.1.0/24
- 10.2.0.102

elastic_agent_endpoint:
- 10.101.1.0/24

manager:
- 172.16.1.185

reciever:
- 172.16.1.188
- 172.16.1.189

searchnode:
- 172.16.1.186
- 172.16.1.187
- 172.16.1.190
- 172.16.1.213
- 172.16.1.214

sensor:
- 172.16.1.191
- 172.16.1.192
- 10.35.1.56

customhostgroup0:
 - 192.168.110.0/24

firewall -> portgroups:
customhostgroup0:
- udp port 2055

firewall -> portgroups -> role -> manager -> chain -> DOCKER-USER -> hostgroups
- 2055

firewall -> portgroups -> role -> manager -> chain -> INPUT -> hostgroups -> customhostgroup0 -> portgroups
- customportgroup0

We guess that it might be a firewall misconfiguration, but any help on how to debug this would be a huge help.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[TheAOWizard]

Found some more info that might help fix this issue by looking in old discussions.

If I do the command sudo curl -sk https://localhost:9200/_cat/indices | sort -n on either the search node or the manager node, I get:

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

If i do sudo so-elasticsearch-pipelines-list on the search node, I get:

[
  "error",
  "status"
]

If I do the same sudo so-elasticsearch-pipelines-list command on the manager node, I get a long list of the pipeline.

I've also tried sudo salt-call pillar.get elasticsearch:auth on both servers, and the credentials are the same on both the search node and the manager node.

The Logstash log on search nodes is saying:

[2024-01-22T00:00:20,310][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error (will retry with exponential backoff) {:code=>500, :url=>"https://so-manager-v01:9200/_bulk", :content_length=>17150, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}"}
[2024-01-22T00:00:20,310][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error (will retry with exponential backoff) {:code=>500, :url=>"https://so-manager-v01:9200/_bulk", :content_length=>15008, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}"}
[2024-01-22T00:00:20,310][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error (will retry with exponential backoff) {:code=>500, :url=>"https://so-manager-v01:9200/_bulk", :content_length=>8574, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}"}
[2024-01-22T00:00:20,311][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error (will retry with exponential backoff) {:code=>500, :url=>"https://so-manager-v01:9200/_bulk", :content_length=>23582, :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}"}

So my new guess is that we don't have firewall misconfiguration, but that we have som DB credential misconfiguration?

[cm-ops]

If you run sudo so-elasticsearch-query _cat/indices, does it ruturn the indices?

[TheAOWizard]

If I run that command on the manager node, it returns the indices.
If I run it on a search node, it returns:
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];","suppressed":[{"type":"master_not_discovered_exception","reason":null},{"type":"master_not_discovered_exception","reason":null}]}],"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];","suppressed":[{"type":"master_not_discovered_exception","reason":null},{"type":"master_not_discovered_exception","reason":null}]},"status":503}

[cm-ops]

From your manager, what does sudo so-elasticsearch-query _cluster/health?pretty show? Verify the search nodes can connect back to the manager'w ES. Also, check node roles with sudo so-elasticsearch-query _cat/nodes.

[TheAOWizard]

We ended up reinstalling everything, 100% clean slate, lost all config and setup, but it's working now.
We threw the old setup in the trash, so we are not able to test anymore on suggestions from this threat, so I'll be closing it now.