so-redis data in /var/lib/docker/volumes/..../_data/

[jpvlsmv]

The so-redis image is configured to have its data in a volume mounted at /data which ends up being a volume in /var/lib/docker/volumes/.../_data/

In my case, that was on a filesystem that wasn't optimal for data storage, and I wanted to put it in /nsm where it seems like it should belong.

I changed it in saltstack/local/redis/init.sls adding a new - /nsm/redis/data:/data to the binds: section, which works, but it would probably be better to create a proper docker_volume (state) and connect it to so-redis properly.

[dougburks]

We primarily use redis as an in-memory buffer and the only data it should write to disk is just in case it needs to reload that data into memory after a crash or shutdown. Since this is really just temporary and transient data, we've never placed it into /nsm.

[jpvlsmv]

Understood, I'm trying to chase down why the temporary and transient data is stuck in my environment, and was trying to eliminate "full root disk" as the problem.

Any other hints on how to troubleshoot large backlogs of both parsed and unparsed records?

The main error message I find is from logstash (I have increased the redis memory limit, and it quickly exhausted the larger limit as well)

[2022-11-11T14:19:30,095][WARN ][logstash.outputs.redis   ] Failed to send backlog of events to Redis {:identity=>"redis://@manager:6379/0 list:logstash:unparsed", :exception=>#<Redis::CommandError: OOM command not allowed when used memory > 'maxmemory'.>

root@manager:/opt/so/log/logstash# so-logstash-get-parsed
(integer) 27422192
root@manager:/opt/so/log/logstash# so-logstash-get-unparsed
(integer) 27425567
root@manager:/opt/so/log/logstash# docker exec -it so-redis redis-cli LLEN logstash:redis
(integer) 0
root@manager:/opt/so/log/logstash# docker exec -it so-redis redis-cli LLEN logstash:parsed
(integer) 0
root@manager:/opt/so/log/logstash# docker exec -it so-redis redis-cli LLEN logstash:unparsed
(integer) 27447182

Thanks,
--Joe

[dougburks]

Any other hints on how to troubleshoot large backlogs of both parsed and unparsed records?

I'd start with the recommendations here:
https://docs.securityonion.net/en/2.3/redis.html#queue