Security Onion should add salt environments functionality

[EbolaWare]

Salt offers the ability to use environments functionality. This would allow SecurityOnion devs and admins to not have to overlay entire files in /opt/so/saltstack/local , which can cause problems on updates. In our case, upgrading to 2.3.130 caused massive issues.

The current configuration utilizes a single salt environment, and prioritizes files in local over default. This is fine if you never update, or do much customization. However, in order to change a single value (that is not configurable via pillar) in a file, you have to copy it from default to local, then make modifications, then in a month when you go to upgrade, you have to remember what and why you changed, or build a diff report before upgrading (if you remember to), to keep track of why you rejigged that file. Then after the upgrade, go through all your changes to see if you broke anything by upgrading.

Using a second environment, prioritized above the base environment, would allow you to just put additions (such as cron jobs or users stuff) into that environment. Think about how you modified your salt/common/init.sls .

Yes, using an include statement would also suffice for this. However, it includes the extra overhead of remembering to update that file at upgrade time.

Please let me know if I'm off base here. I've been complaining, and wanted to get other users'/admins' opinions on this. Especially if you have a large grid or highly customized environment.

[dougburks]

soup should have notified you that you have local configurations. Did you not see see it in the output?

From https://docs.securityonion.net/en/2.3/soup.html#local-configurations:

soup will check for local configurations in /opt/so/saltstack/local/ that may cause issues and flag them with the message Potentially breaking changes found in the following files. Please examine the output of soup and review any local configurations for possible issues.

[darth-drew]

I've seen a number of posts where I believe this approach would also have saved some heartaches, including #8385 and #7674. I can do an exhaustive search when I have more time if that would be helpful to the SO maintainers. To add to this, our custom watermark settings stopped being honored post upgrade, resulting in the third unique outage post upgrade to .140 without being warned of potentially breaking changes. From my perspective, implementing salt environments would save countless hours of troubleshooting after upgrades (I'm personally in the 100s) and future SO discussion posts (of which I have already posted a few too many WRT upgrades). This would be a huge benefit to the community, and a reduction in time required for support by the SO team, freeing you to continue adding other cool features to the product.

I'll get off my soapbox. Thanks for all the hard work!

[dougburks]

@EbolaWare

You are right, we should have seen that, but after grepping through soup.log, none were found.

I've added the following to https://docs.securityonion.net/en/2.3/soup.html#elastic-8:
Before upgrading, you should review any local modifications that you’ve applied in the past. In particular, some folks have experienced issues with local Filebeat modifications. For example, if you have the file /opt/so/saltstack/local/salt/filebeat/init.sls, then you should move or rename it before beginning the upgrade. After the upgrade, you can copy the new /opt/so/saltstack/default/salt/filebeat/init.sls to /opt/so/saltstack/local/salt/filebeat/init.sls and then re-apply your local modification.

I've also created the following issue so that soup checks for local configurations before modifying the system:
#8423

I'm likely going to push for our team to roll this out, and if there are enough users interested, I'll document the process, and attempt to get it pushed here.

You're of course welcome to try Salt environments and let us know how it goes.

In the same light, I'd like to give recommendations on updating your saltstack documentation section with an entry on managing large installations and customizations.

Please feel free to submit pull requests for the documentation.

in the case of an unexpected (or unfortunately necessary) reboot or a power loss event. This may have been part of our issue

Power outages can cause lots of problems, not just during soup. For that reason, we recommend UPS:
https://docs.securityonion.net/en/2.3/hardware.html#ups

[dougburks]

@darth-drew

countless hours of troubleshooting after upgrades (I'm personally in the 100s)

You seem to be experiencing more issues than most of our users. We don't want you to have a bad experience with Security Onion but it's hard for us to know exactly what all you have customized and why. We want to help you succeed. The quickest and easiest way for us to help you is via Professional Services. I'm not trying to be a salesman here, just trying to provide some ammunition for your management chain to hopefully save you time and effort so that you can focus on catching bad guys! 😄

[darth-drew]

Thanks Doug. Totally understand you can't account for all customizations, of which we have made quite a few.

We started the process in March to get SO PS purchased. Unfortunately, although approved at a high level, the purchase has turned into a paperwork nightmare on the administrative side causing numerous delays. Pete and I have been patiently waiting for something to shake loose from my company.

[dougburks]

Yep, I'm tracking those discussions, thus my use of management chain above! 😉

[m0duspwnens]

While we utilize only one Salt environment in Security Onion, it is almost functioning the same as mulitple environments.

Below, is how we have configured the file_roots within /etc/salt/master and the top file located at /opt/so/saltstack/default/salt/top.sls. The file_roots dictates the order in which the Salt master will search for the file. If the file is found in local, then it will not look for the file within default. The top file determines which node, get which states and from which environment it is pulled from.

/etc/salt/master

file_roots:
  base:
    - /opt/so/saltstack/local/salt
    - /opt/so/saltstack/default/salt

/opt/so/saltstack/default/salt/top.sls : note that this example has been simplified

base:
  '*':
    - cron.running
    - repo.client

So from the top file, we can see that all nodes * will process a file located in the cron directory named running.sls. They will also process a file called client.sls located in the repo directory. It will find these by first checking if they exist in /opt/so/saltstack/local/salt and if it doesn't, it will then check in /opt/so/saltstack/default/salt.

If we consider trying to use multiple environments, then we modify /opt/so/saltstack/default/salt/top.sls to look like the following:

local:
  '*':
    - cron.running
    - repo.client

base:
  '*':
    - cron.running
    - repo.client

Now that we have two environments, we need to tell the Salt master about the other environment location, we do this by modifying the the config at /etc/salt/master:

file_roots:
  base:
    - /opt/so/saltstack/default/salt
  local:
    - /opt/so/saltstack/local/salt

So now, if we apply the cron.running state to a node, it will first check in local environment located at /opt/so/saltstack/local/salt and if the file isn't found there, it will try to find it in the base environment located at /opt/so/saltstack/local/salt. This is functionally the same as what we are currently doing.

A good example of what multipe environments are used for is a dev, qa, prod configuration as seen here: https://docs.saltproject.io/en/latest/ref/states/top.html#multiple-environments

I hope I was able to help with the question here. If there are any further questions, please feel free to ask.