No data in dashboard and Elastic Agents not sending data

[2nutz4u]

Version

2.4.90

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

4

RAM

32

Storage for /

293GB

Storage for /nsm

700G

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I have experience in stalling SO as I already have one in production but want to perform a new setup as lots of changes have been made with version 2.4. I have a distributed install 1 master and 2 search nodes. So-status shows everything running and healthy.

Security Onion Status
Container │ Status │ Details
───────────────────────────────────┼─────────┼────────────────────────────
so-dockerregistry │ running │ Up About an hour
so-elastalert │ running │ Up 14 minutes
so-elastic-fleet │ running │ Up About an hour
so-elastic-fleet-package-registry │ running │ Up About an hour (healthy)
so-elasticsearch │ running │ Up 17 minutes
so-idstools │ running │ Up About an hour
so-influxdb │ running │ Up About an hour (healthy)
so-kibana │ running │ Up 16 minutes
so-kratos │ running │ Up About an hour
so-logstash │ running │ Up 15 minutes
so-nginx │ running │ Up About an hour (healthy)
so-redis │ running │ Up About an hour
so-sensoroni │ running │ Up About an hour
so-soc │ running │ Up About an hour
so-telegraf │ running │ Up About an hour
I have nothing in the dashboard. I have installed couple of Elastic agents (yes installed from the downloads section with no errors) but no logs are being collected. Nothing in discovery all empty.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[2nutz4u]

tail logstash.log
] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}

[bushcraftbuddy]

Do you have any forward nodes? this issue usually happens when your manager can't send events to a search node. So it's odd that you are having this issue with two of them connected

but I am not understanding how you have your Network traffic being monitored. are you attempting to monitor using the search nodes?

Could you send a Screenshot of your grid (without IP address or hostnames)?

[2nutz4u]

I do not have a forwarder setup right now. If I understand correctly from my previous installation of SO, I should be at least getting logs from the agents. Also, I performed a new install same issue. Same error message in logstash.

[JhonShell]

Hi 2nutz4u,
Could you please elaborate more about your deployment so we can help you, example: distribute one managersearch and receiver , etc.

[2nutz4u]

Sure. I am running distributed model. One manager and two search node.

[JhonShell]

Hi,

please check the firewall rule and make sure that the traffic is coming to the manager with tcpdump port 5055 for logstash data and 8220 for the fleet manager server, if all of this is correct you should start troubleshooting further.

[2nutz4u]

Firewall looks good see below.
tcpdump 5055

tcpdump 8220

[2nutz4u]

I tried fixed mention here (#12074) did not solve my issue

[JhonShell]

Please now check the log of Logstash at /opt/Logstash log, to check if there is any related error.

[2nutz4u]

[root@svr-securityonion logstash]# tail logstash.log
[2024-08-29T18:03:08,502][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:08,502][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:09,505][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:09,505][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:10,509][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:10,509][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:11,512][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:11,513][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:12,516][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}
[2024-08-29T18:03:12,516][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://@svr-securityonion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}

[JhonShell]

Hi , please check the configuration make sure that the entry is created

is the agent of the manager reflected on the fleet server.

[2nutz4u]

Yep, looks correct to me. Also, the agents are sowing up as healthy.

[bushcraftbuddy]

when you go to Administration/Configuration/Firewall/hostgroups/searchnode: do you have the Ip address of your search nodes put in there?

If not add them and look at your Redis queue size on your manager in the grid page, it should be going down slowly, and you should start to see things show up in the dashboard.

[2nutz4u]

Yes, I do . I could be wrong but they were required when joining the search nodes. Best Regards, Rakesh Patel [image: Bhollo Corp] <https://www.facebook.com/bhollocorp> [image: Bhollo Corp] <https://twitter.com/bhollocorp>[image: https://plus.google.com/+Bhollocorp] <https://plus.google.com/+Bhollocorp>

…

On Thu, Aug 29, 2024 at 5:37 PM bushcraftbuddy ***@***.***> wrote: when you go to Administration/Configuration/Firewall/hostgroups/searchnode: do you have the Ip address of your search nodes put in there? If not add them and look at your Redis queue size on your manager in the grid page, it should be going down slowly, and you should start to see things show up in the dashboard. — Reply to this email directly, view it on GitHub <#13567 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHIJ23EO4SWJPUPGO2AAWG3ZT6PCHAVCNFSM6AAAAABNIWJDISVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBZGIZTENI> . You are receiving this because you authored the thread.Message ID: <Security-Onion-Solutions/securityonion/repo-discussions/13567/comments/10492325 @github.com>

[2nutz4u]

Providing pictures just in case.

The count is the same and has not gone down.

[2nutz4u]

Performed new install with the latest build. All working as advertised. Thanks everyone for their assistance.

[JhonShell]

Hi there, now I am having the same issue than you after fixing previous issue.

[2nutz4u]

I don't follow. Can you elaborate?

[JhonShell]

now my receiver is throwing this error :[2024-09-06T20:02:21,730][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis
ception=>#<Redis::CannotConnectError: Error connecting to Redis on

[2nutz4u]

I was never able to figure out why. I gave up on the 2.4.90 install and downloaded the latest 2.4.100. I performed a new install instead of the upgrade and everything is working. If I had to guess it is broken logstash pipe is broken. My working version does not show the same error. Sorry could not be more helpful.