Standalone Receiver

[argwfm]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

256

Storage for /nsm

256

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Are receiver nodes expected to sustain operations when the manager is offline?

Based on limited testing, when the manager is offline, the search nodes stop draining the receiver redis queue, redis eventually fills, and logstash stops processing until the manager is back online. The same does not occur for the manager if/when the receiver is offline. It appears the manager is required to coordinate elasticsearch cluster operations.

Considering the below data (counts of SO log output), when the receiver (red) is offline between ~16:00- 16:15 the manager (blue) continues processing events w/out issue. When this is reversed and the manager is offline between ~16:30- 16:40, the receiver stops once redis maxmemory is exhausted; only after the manager is back online do the search nodes begin draining redis again.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[TOoSmOotH]

Can you add a master role to one of your search nodes and try this exercise again? My guess is that the cluster went to red because there was no nodes with the master role. You could find that as well in the logs for the search nodes during that period.

[argwfm]

Loss of master appears to be the root cause based on the logs paired w/ no other master candidates.

Is there a preferred way to issue PUT commands against the cluster to config a secondary master candidate, so-elasticsearch-query w/ an argument? Will also need to update /etc/hosts so the candidate master can be resolved by name.

[WARN ][org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper] master not discovered yet: have discovered [{manager}{}{}{search1}{search1}{search1:9300}{dhit}{8.10.4}{7000099-8100499}]; discovery will continue using [manager:9300] from hosts providers and [{manager}{}{}{manager}{manager}{manager:9300}{dmrt}{8.10.4}{7000099-8100499}] from last-known cluster state; node term 10, last-accepted version 5225 in term 10; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.10/discovery-troubleshooting.html

[TOoSmOotH]

Just add the role in the config gui. Make sure you open advanced and you will see where roles can be set per node. As far as hosts files we don't manage those but we might be able to add extra hosts for all nodes in a future release. For now would be manual.

[argwfm]

That fixed it, thanks much @TOoSmOotH!

config> elasticsearch> so_roles> so-searchnode> config> node> roles: master

[INFO ][org.elasticsearch.cluster.coordination.Coordinator] master node [{manager}{}{}{manager}{manager}{manager:9300}{dmrt}{8.10.4}{7000099-8100499}] disconnected, restarting discovery
[INFO ][org.elasticsearch.cluster.service.ClusterApplierService] master node changed {previous [], current [{search1}{}{}{search1}{search1}{search1:9300}{dhimt}{8.10.4}{7000099-8100499}]}, term: 16, version: 9307, reason: ApplyCommitRequest{term=16, version=9307, sourceNode={search1}{}{}{search1}{search1}{search1:9300}{dhimt}{8.10.4}{7000099-8100499}{transform.config_version=10.0.0, xpack.installed=true}}

[argwfm]

Circling back on this... adding candidate master nodes resolved the cluster going red when the manager is unavailable, but did not completely resolve this issue (redis stops draining). Per the present search node logstash pipeline 9805_output_elastic_agent.conf.jinja output is bound to the manager, so this is appearing as another SPoF.

[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://so_elastic:xxxxxx@manager:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://manager:9200/][Manticore::SocketException] No route to host"}

Should the candidate master nodes be included here to resolve?

/opt/so/saltstack/default/salt/logstash/pipelines/config/so/9805_output_elastic_agent.conf.jinja
output {
  if "elastic-agent" in [tags] {
    if [metadata][pipeline] {
      if [metadata][_id] {
        elasticsearch {
          hosts => "{{ GLOBALS.manager }}"
          ecs_compatibility => v8
          data_stream => true
          user => "{{ ES_USER }}"
          password => "{{ ES_PASS }}"
          document_id => "%{[metadata][_id]}"
          pipeline => "%{[metadata][pipeline]}"
          silence_errors_in_log => ["version_conflict_engine_exception"]
          ssl => true
          ssl_certificate_verification => false
        }
      }
      else {
        elasticsearch {
          hosts => "{{ GLOBALS.manager }}"
          ecs_compatibility => v8
          data_stream => true
          user => "{{ ES_USER }}"
          password => "{{ ES_PASS }}"
          pipeline => "%{[metadata][pipeline]}"
          ssl => true
          ssl_certificate_verification => false
        }
      }
    }
    else {
      elasticsearch {
        hosts => "{{ GLOBALS.manager }}"
        ecs_compatibility => v8
        data_stream => true
        user => "{{ ES_USER }}"
        password => "{{ ES_PASS }}"
        ssl => true
        ssl_certificate_verification => false
      }
    }
  }
}

[TOoSmOotH]

Yea this would be a problem. We are looking into it now.

[TOoSmOotH]

@argwfm Does your cluster have a quorum for the master? What we are observing is stops ingesting after a bit due to no quorum.

[argwfm]

Not from what I can see... search1 is currently elected master, when elasticsearch is stopped on the manager, not much changes except the manager is removed from the registered nodes across the cluster. No abnormalities present w/in elasticsearch.log over the duration; several errors w/in logstash.log, as referenced above w/ no route to the manager. LMK if I can provide additional details/logging.

Before stopping elasticsearch on manager:

search2# so-elasticsearch-query _cat/master
********************** search1 172.x.x.1 search1

search2# so-elasticsearch-query _cat/nodes
172.x.x.1  8 94 17 1.45 1.68 1.70 dhimt * search1
172.x.x.10 39 93 19 2.66 2.33 2.25 dmrt  - manager
172.x.x.4 10 99 22 1.92 1.88 1.75 dhit  - search4
172.x.x.2 25 99 21 1.36 1.13 1.23 dhimt - search2
172.x.x.3 49 99 16 0.75 1.33 1.42 dhit  - search3

After stopping elasticsearch on manager:

search2# so-elasticsearch-query _cat/master
********************** search1 172.x.x.1 search1

search2# so-elasticsearch-query _cat/nodes
172.x.x.1 58 99 7 1.39 1.32 1.17 dhimt * search1
172.x.x.4 57 99 7 1.21 1.51 1.45 dhit  - search4
172.x.x.2 19 98 3 0.60 1.06 1.24 dhimt - search2
172.x.x.3 40 99 4 1.14 1.59 1.73 dhit  - search3

Relevant elasticsearch.log from search2:

[2023-12-13T19:35:58,821][INFO ][org.elasticsearch.cluster.service.ClusterApplierService] removed {{manager}{}{}{manager}{manager}{172.x.x.10:9300}{dmrt}{8.10.4}{}}, term: 56, version: 15609, reason: ApplyCommitRequest{term=56, version=15609, sourceNode={search1}{}{}{search1}{search1}{172.x.x.1:9300}{dhimt}{8.10.4}{}{xpack.installed=true, transform.config_version=10.0.0}}

[2023-12-13T19:38:11,770][INFO ][org.elasticsearch.cluster.service.ClusterApplierService] added {{manager}{}{}{manager}{manager}{172.x.x.10:9300}{dmrt}{8.10.4}{}}, term: 56, version: 15612, reason: ApplyCommitRequest{term=56, version=15612, sourceNode={search1}{}{}{search1}{search1}{172.x.x.1:9300}{dhimt}{8.10.4}{}{xpack.installed=true, transform.config_version=10.0.0}}

[m0duspwnens]

This issue will be fixed in 2.4.40: #12037

[argwfm]

Thank you, all!