How soup updates work in DISTRIBUTED mode

[TOoSmOotH]

I wanted to take a minute and discuss how soup works for distributed mode. I think there might be some confusion on this topic and how it differs from the previous version of Security Onion. When you run soup on the manager here is the process:

1. Checks to see if this is being run on a manager.
2. Checks to see if you are in airgap mode. If the grid is in airgap mode you will get prompted for the location of the ISO or mount point.
3. soup then checks if it is running the latest version of soup. If you are not running the latest it will put the latest in the correct place and ask you to re-run soup.
4. Compares the installed version with what is available on github/ISO.
5. Checks to see if salt needs to be updated. More on this later.
6. Pulls down all the new docker containers or if airgap, copies them from the ISO.
7. Stops the salt master and minion and restarts it in a restricted mode. This mode only allows the manager to connect to it so that we make sure the manager is done before any of the minions are updated.
8. IF salt needs to update it happens now. This will cause the master and minion services to restart but still in restricted mode.
9. soup now makes any changes to pillars that are needed such as adding new settings or renaming values. This varies from release to release.
10. If you are in airgap mode it now copies the latest ET Open rules as well as yara rules to the manager.
11. The new salt code is put into place on the manager.
12. If you have fleet enabled, it re-generates new osquery packages.
13. Runs a highstate on the manager which is the actual upgrade where it will use the new salt code and docker containers.
14. soup will now unlock the salt master service and allow minions to connect again.
15. Issues a command to all minions to update salt if salt needs to be updated. This is important to note as it takes time to to update the salt minion on all minions. If the minion doesn't respond for whatever reason, it will not be upgraded at this time. This is not an issue because the first thing that gets checked when a minion talks to the master is if salt needs to be updated and will apply the update if it does.
16. Nodes connect back to the manager and actually perform the upgrade to the new version.

This part is important if you did not take the time to read the wall of text above. Just because the update completed on the manager does NOT mean the upgrade is complete. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is even more important if you are using true clustering for Elastic.

Each minion is on a random 15 minute check in period and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. This will definitely be the case when upgrading to 2.3.40 because of the way the basic license SSL works.

Before you go restarting any services because you are missing data, please check and make sure at least one search node has completed its upgrade. The best way to do this is to run sudo salt-call state.highstate from a search node and make sure there are no errors. Typically if it works on one node it will work on the rest. Forward nodes are less complex and will update as they check in so you can monitor those from the grid section of SOC.

I know to some users this seems a little strange, but imagine you have 20 search nodes and 400 forward nodes. You do not have the time to update each one individually, which is why soup operates the way it does. The way you manage 2 search nodes and 4 forward nodes is the same way you would do 20/400 which allows you to spend more time finding evil than administrating the grid.

[clairmont32]

Thank you for describing this lengthy process!

[dougburks]

We've also added this information to our documentation:
https://docs.securityonion.net/en/2.3/soup.html#distributed-deployments

[palichx]

Hello,
may be need check system updates? If it doesn't have last packet "docker-cli" services will not started successfully on minions. it's looks like:
Checking Docker status

Docker --------------------------------------------------------------- [ OK ]    

Checking container statuses

so-filebeat -------------------------------------------------- [ WAIT_START ]    
so-nginx ----------------------------------------------------- [ WAIT_START ]    
so-sensoroni ------------------------------------------------- [ WAIT_START ]    
so-steno ----------------------------------------------------- [ WAIT_START ]    
so-strelka-backend ------------------------------------------- [ WAIT_START ]    
so-strelka-coordinator --------------------------------------- [ WAIT_START ]    
so-strelka-filestream ---------------------------------------- [ WAIT_START ]    
so-strelka-frontend ------------------------------------------ [ WAIT_START ]    
so-strelka-gatekeeper ---------------------------------------- [ WAIT_START ]    
so-strelka-manager ------------------------------------------- [ WAIT_START ]    
so-suricata -------------------------------------------------- [ WAIT_START ]    
so-telegraf -------------------------------------------------- [ WAIT_START ]    
so-wazuh ----------------------------------------------------- [ WAIT_START ]    
so-zeek ------------------------------------------------------ [ WAIT_START ]  

Thanks!