Heavynode Logstash EPS at 0

[ben-sec]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128 GB

Storage for /

222 GB

Storage for /nsm

11 TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hi!
Yesterday I have setup an new heavynode for my new SO 2.4.60 distributed environment. Today I saw the error "Logstash EPS at 0" on the InfluxDB status page. It looks like the Logstash pipeline is broken on my heavynode. I checked logstash.log and found the following warnings and errors:

[2024-04-26T06:34:42,763][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "ssl" set in elasticsearch.
[2024-04-26T06:34:42,787][WARN ][logstash.outputs.elasticsearch] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set `ssl_verification_mode => full`
[2024-04-26T06:34:43,471][ERROR][logstash.inputs.beats    ] SSL configuration invalid {:exception=>Java::JavaLang::IllegalArgumentException, :message=>"File does not contain valid certificates: /usr/share/logstash/elasticfleet-logstash.crt", :cause=>{:exception=>Java::JavaSecurityCert::CertificateException, :message=>"could not find certificate file: /usr/share/logstash/elasticfleet-logstash.crt"}}
[2024-04-26T06:34:43,552][ERROR][logstash.javapipeline    ] Pipeline error {:pipeline_id=>"manager", :exception=>#<LogStash::**ConfigurationError: File does not contain valid certificates**: /usr/share/logstash/elasticfleet-logstash.crt>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-beats-6.6.4-java/lib/logstash/inputs/beats.rb:384:in `new_ssl_handshake_provider'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-beats-6.6.4-java/lib/logstash/inputs/beats.rb:247:in `create_server'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-beats-6.6.4-java/lib/logstash/inputs/beats.rb:242:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1987:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipelines/manager/0011_input_endgame.conf", "/usr/share/logstash/pipelines/manager/0012_input_elastic_agent.conf", "/usr/share/logstash/pipelines/manager/0013_input_lumberjack_fleet.conf", "/usr/share/logstash/pipelines/manager/9999_output_redis.conf"], :thread=>"#<Thread:0x15316991 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

It's a plain vanilla heavynode installation and I have not changed a thing. The rest of my SO deployment (manager+search+sensor) works fine. Any ideas on how to fix the logstash pipeline on the heavynode? Thanks in advance!

Cheers, Ben

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Have you looked at /etc/pki/elasticfleet-logstash.crt to see what the issue is with the certificate?

[ben-sec]

f I try openssl x509 -in elasticfleet-logstash.crt -text -noout on the heavynode I just get

Could not read certificate from elasticfleet-logstash.crt
Unable to load certificate

This works fine on my manager node and the certificate is valid.

[ben-sec]

Can I somehow recreate this broken certificate?

[ben-sec]

The Security Onion Pro Service explained to me, that the error "Logstash EPS at 0" on the InfluxDB status page is a bug in case of heavynodes. Everything works fine, alarms show up in SOC, but you will not see data in Kibana that is stored in the Elasticsearch instance on a heavynode (yet).