No longer receive alerts on the website

[tthomasb]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

70 GB

Storage for /nsm

130 GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

Hello there,

after the weekend, I saw /nsm was nearly full, so I delete some pcap files (auto delete doesn't work well) and deactivate Stenographer (just using suricata). But since the influxdb show "NSM Disk High Usage" the traffic doesn't show as Altert to the managersearch. Before the weekend, everything works fine.

Already tried after searching similar issue:
so-status: Everything fine
salt * so.status: Everything fine
salt * state.highstate: Everything fine
Traffic comes to sensor: Yes
Traffic comes to manager (tcpdump): Yes
Restart server and logstash
Check firewall: Everything fine

Every log file looks fine except of this logstash:

logstash.log

[2024-02-12T10:10:14,290][WARN ][logstash.outputs.redis ] Failed to flush outgoing items {:outgoing_count=>125, :exception=>"Redis::CommandError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]} [2024-02-12T10:10:14,290][WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis {:identity=>"redis://<password>@seconion:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1587:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in block in multi_receive'", "org/jruby/RubyArray.java:1987:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:104:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}

Any idea whats going on?

Kind regards

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[InfosecGoon]

Do you get any errors when you run "sudo so-redis-restart" on the Manager Search?

[tthomasb]

Thanks for your fast reply!

No Errors and no Changes... The only change is some kind of weird (see below), but I think this shouldn't be the problem.

I just use the default settings and IP range from SOC.

[INFO ] {'container': {'Networks': {'bridge': {'IPConfiguration': {'old': 'automatic', 'new': 'not connected'}}, 'sobridge': {'Aliases': {'old': None, 'new': ['XXXX']}, 'IPAMConfig': {'old': None, 'new': {'IPv4Address': '172.17.1.33'}}}}}, 'container_id': {'added': 'XX'}, 'state': {'old': None, 'new': 'running'}}

[reyesj2]

The installation type says distributed, but is this a standalone node?
What is the output of
df -h /nsm && du -h -d 1 /nsm

[tthomasb]

Hey, thanks for the reply!

I definitely installed it as a distributed system with one managersearch and one sensor. It works fine for a few days. I have tested it with a custom ping rule, which works fine. At the moment the sensor is logging this rule but not showing it in the alerts on the managersearch...

Output Managersearch: df -h /nsm && du -h -d 1 /nsm

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-nsm 124G 31G 93G 25% /nsm
6.7G /nsm/docker-registry
10G /nsm/repo
2.8G /nsm/elastic-fleet
66M /nsm/rules
0 /nsm/soc
0 /nsm/pcap
0 /nsm/import
0 /nsm/pcapout
6.3G /nsm/backup
2.2G /nsm/elasticsearch
4.0K /nsm/logstash
152M /nsm/redis
204M /nsm/mysql_old_1706798946
588K /nsm/kratos_old_1706798946
14M /nsm/influxdb_old_1706798946
206M /nsm/mysql_old_1706801944
588K /nsm/kratos_old_1706801944
13M /nsm/influxdb_old_1706801944
232M /nsm/mysql_old_1706855972
588K /nsm/kratos_old_1706855972
82M /nsm/influxdb_old_1706855972
206M /nsm/mysql_old_1706863494
588K /nsm/kratos_old_1706863494
39M /nsm/influxdb_old_1706863494
206M /nsm/mysql_old_1706865919
588K /nsm/kratos_old_1706865919
13M /nsm/influxdb_old_1706865919
193M /nsm/mysql_old_1706867779
588K /nsm/kratos_old_1706867779
9.6M /nsm/influxdb_old_1706867779
748K /nsm/kratos
379M /nsm/influxdb
255M /nsm/mysql
30G /nsm

Output Sensor: df -h /nsm && du -h -d 1 /nsm

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-nsm 312G 15G 297G 5% /nsm
6.3G /nsm/docker-registry
2.6G /nsm/repo
1.4G /nsm/elastic-fleet
92M /nsm/pcap
0 /nsm/import
0 /nsm/pcapout
0 /nsm/pcaptmp
1.3M /nsm/pcapindex
1.8G /nsm/suricata
303M /nsm/strelka
13G /nsm

[tthomasb]

I have found the problem. The Redis queue was full. The solution can be found in the official documentation: https://docs.securityonion.net/en/2.4/redis.html#queue

Many thanks for your help.