Search Manager node not passing from redis to Elasticsearch

[jcmadick]

SO Version 2.3.160. Virtualized Search Manager, 5 physical heavy nodes. On prem, with Internet access. Installed from ISO. All services running on the manager, no failures running state.highstate. No grid failures.

The issue is that on the search manager node, we can see information being ingested into logstash (confirmed from both the client and the search manager node.) We can see the information flow into redis in the logstash log. Redis count just climbs, never goes down and eventually we get the OOM command error in the logstash log. Looking on elasticsearch on the search manager, we don't see indices for any so queues other than so_playbook and so_cases. The heavy nodes operate fine, and the only reason we found this issue is because we started deploying winlogbeats to DC's. Global.sls (de-identified attached.)

We've looked at everything we can think of, so any idea where we should be looking to get elastic talking to redis would be appreciated.

global.sls.txt
foleyonionmanager.log
redis-server.log

[dougburks]

When you configured your manager node, did you configure it as a manager or managersearch?

It sounds like you configured it as just as a manager.

From https://docs.securityonion.net/en/2.3/architecture.html#distributed:

If you install a dedicated manager node, you must also deploy one or more search nodes. Otherwise, all logs will queue on the manager and have no place to be stored. If you are limited on the number of nodes you can deploy, you can install a manager search node so that your manager node can act as a search node and store those logs. However, please keep in mind that overall performance and scalability of a manager search node will be lower compared to our recommended architecture of dedicated manager node and separate search nodes.

Heavy nodes do not consume from the Redis queue on the manager. This means that if you just have a manager and heavy nodes, then the Redis queue on the manager will grow and never be drained. To avoid this, you have two options. If you are starting a new deployment, you can make your manager a manager search so that it will drain its own Redis queue. Alternatively, if you have an existing deployment with a manager and want to avoid rebuilding, then you can add a separate search node (NOT heavy node) to consume from the Redis queue on the manager.

[jcmadick]

They said it was configured as a managersearch. Is there a way to verify the initial installation choice?

[dougburks]

What is the output of the following?

sudo cat /opt/so/saltstack/local/pillar/data/managersearchtab.sls

[jcmadick]

That was it, it was actually just set up as a manager. Thanks