HELP Please - Failed to send backlog of events to Redis

[jrmsq]

Good afternoon,
I just added a 1.5TB to my manager to give more room to the /nsm folder. After this process now I have no alerts in SOC and also, Kibana is not taking my login. Can someone help or point me in the right direction?

[WARN ][logstash.outputs.redis ] Failed to send backlog of events to Redis
{:identity=>"redis://@soc-mngr:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/redis-4.8.1/lib/redis/client.rb:162:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/redis-4.8.1/lib/redis.rb:270:in block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/redis-4.8.1/lib/redis.rb:269:in send_command'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/redis-4.8.1/lib/redis/commands/lists.rb:86:in rpush'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:152:in flush'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/buffer.rb:221:in block in buffer_flush'", "org/jruby/RubyHash.java:1519:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:209:in send_to_redis'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-codec-json-3.1.1/lib/logstash/codecs/json.rb:69:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:74:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:68:in time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-redis-5.0.0/lib/logstash/outputs/redis.rb:123:in receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in block in multi_receive'", "org/jruby/RubyArray.java:1865:in each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in block in start_workers'"]}

[InfosecGoon]

Can you tell us how you added the new disk to the Manager? Did you follow these directions?

https://docs.securityonion.net/en/2.3/new-disk.html

[jrmsq]

Thank you @InfosecGoon for replying and yes, I used that one.

[InfosecGoon]

Try bumping up the value of redis_maxmemory in /opt/so/saltstack/local/pillar/global.sls, and then restart redis with so-redis-restart. I don't know how much memory you have in this box, but even doubling the value should help.

[jrmsq]

Thank you, mine was on 830, I gave it 2Gs, still nothing. This is what I get when I get to the SOC. Security Onion Request failed with status code 500 See the Help section of the Security Onion documentation for additional troubleshooting guidance.

[EbolaWare]

Looks like your SOC can't get a reply from redis because redis is full. (I've been wrong before...)

I've experienced this, and it is frustrating. I'm going to assume this is a distributed cluster.

1. Go to the master or a receiver node. Run the so-redis-count (or whatever it is) twice. If the numbers don't change, and it's not 0, it's stuck.
2. Make sure you have no locked indexes in elastic (your indices are probably not ingesting either). And you've got disk space on the search nodes.
3. Check to see if logstash on the search nodes is sending events to elastic. (It's probably not) so-logstash-events/pipeline-stats or something like that... Run twice, check if there are differences.

I'm assuming everything is locked up at this point... On every node running redis, do docker stop so-redis (running so-restart-<whatever> will destroy the container and its contents and I don't like losing millions of events...)
Go to the search nodes and restart the logstash containers salt '*searchnode' cmd.run cmd='docker restart so-logstash'and then start up your redis container(s) again docker start so-redis.

The redis queue should begin to drain, and new events come in. If new events aren't coming in, you'll also need to restart the logstash on manager and receivers. If events still won't go into elastic, but the cluster is healthy, you've got an issue with an index. There are ways to unlock them, but it seems to be faster to blow out the problem index and replay the data later.

Hopefully this is helpful and not completely off base. Other suggestions included bumping up redis max memory. Definitely a good idea!
However, creating an alert in grafana based on a 25-50% watermark of events in this would be a better idea (send a message to slack or email if you're not airgapped). That way you have time to log in and fix it before it locks up.

Good luck!

[jrmsq]

Thank you @EbolaWare , I have cleared out the redis before following the instructions found there: https://docs.securityonion.net/en/latest/redis.html.

I have 2Gs mem assigned to it and it was at 850mb. Still nothing.

[EbolaWare]

Is anything flowing into elastic cluster?

[jrmsq]

Caused by: org.elasticsearch.action.NoShardAvailableActionException: [soc-mngr][192.168.6.110:9300][indices:data/read/search[phase/query]]
[2023-07-06T15:27:11,061][INFO ][org.elasticsearch.cluster.routing.allocation.AllocationService] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[elastalert_status_status][0]]])." previous.health="RED" reason="shards started [[elastalert_status_status][0]]"

is what I am seeing in the elasticsearch logs. I tried restarting so-elasticsearch on my search node and it took almost 5 minutes, then I decided to reboot the server. I'll keep you posted.

[jrmsq]

Getting the same error message on all nodes with elasticsearch running.

this node is locked into cluster UUID [X7s7DgVeR2qKNbSilQuSBQ] but [cluster.initial_
master_nodes] is set to [snode2]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.7/important-settings.html#initial_master_nodes