2.3.2 Upgrade so-soc Error

[jhildreth]

Hello!
First of all - thanks so much to the Security Onion team for all your work - I'm really impressed with SO2!
I had a little trouble with my grid after upgrading from 2.3.0 to 2.3.2. I primarily attributed it to an issue with ternary shortcuts in the Jinja template for soc.json.

I actually saw #1653 after writing this all up, and I would imagine this may be a duplicate of that? I still thought it safer to start a new discussion, since I'm having a bit of what seems to be a related issue with the Redis queue. I'm sorry if it's a complete duplicate.

Version: 2.3.2 (upgraded from 2.3.0)
Install Source: network install on Ubuntu 18.04
Install Type: Distributed (1 manager, 1 search, 1 forward)

so-status shows all "OK" on all 3 nodes - though over the course of this issue so-soc was "ERROR" and then "MISSING" on the manager node.
sudo salt-call state.highstate succeeds on manager, fails with "Data failed to compile:" (and nothing listed) on search node (I guess I'm not supposed to run it on other than the manager node).

I upgraded the grid yesterday afternoon, by running sudo soup on the manager node. It completed successfully, and while it did not need to be run a second time, I did just to confirm - and it indicated everything was up-to-date at 2.3.2. I noticed the web ui for SOC still showed 2.3.0, which I thought was odd. (I actually wondered along the same lines as #1670, if there were some way for me to look and see what version was actually running on each node.)

I stopped working with the system shortly after upgrading, but this morning I found that the Redis queue on the manager had continued to grow, and ultimately the machine had locked up. I rebooted it, ssh'd in, and saw the message that the search node needed to be rebooted to update, so I did that.

At this stage, sudo salt \* so.status showed everything "OK" except for so-soc on the manager, which showed "ERROR". so-soc-restart would fail, with the comment for socconfig: "Unable to manage file: Jinja variable the inline if-expression on line 36 evaluated to false and no else section was defined."

Now so-soc showed as "MISSING". I ran so-docker-refresh, which did pull down some new layers (which surprised me). so-soc-restart still resulted in the same error, though.

As an experiment, I modified line 36 in /opt/so/saltstack/default/salt/soc/files/soc/soc.json from
"https://{{ SN.split('_')|first }}:9200"{{ "," if not loop.last }}
to
"https://{{ SN.split('_')|first }}:9200"{{ "," if not loop.last else ""}}
This allowed so-soc-restart to succeed, and all statuses were "OK" (I also noticed the web ui showed 2.3.2 at this point). The "remoteHostUrls" was now in the "elastic" section of /opt/so/conf/soc/soc.json (it had not been before). I noticed it was using the hostname for my search node, which my manager node did not have a way to resolve, so I added that to /etc/hosts on the manager.

At this stage, it seemed like everything should be working correctly, but the Redis queue on the manager has stayed high, and load on the search node was still very high. I didn't find anything particularly interesting in the redis logs on the manager, or the elasticsearch logs on the manager or search node, and rebooting both the manager and search nodes did not help. However, calling state.highstate on the manager (for starting this discussion) did change a few things (including so-elasticsearch-templates and crossclusterson), and now the Redis queue is finally coming down on the manager.

I'm still not 100% sure there isn't another issue I'm wrestling with here, but it seems like the only problem for me may have been the ternary shortcut in the Jinja template. I suspect that would have been an issue for anyone with items in nodestab, right? If so, I'm happy to open an issue (and maybe a PR) if that would be helpful.

[TOoSmOotH]

I suspect this is Ubuntu?

[jhildreth]

Yes, this is a network install on Ubuntu 18.04.

[jhildreth]

And I just saw the answer on #1653, that this already has a fix merged in. :) As I had said, I wanted to open the issue in case there was more going on with my install - but the Redis queue has finally worked back down to zero, so I think I'm all set.

Thanks again for all your work!