Is so-saltstack-update safe to use for SO v2.4?

[DoughnutPeach]

Version

2.4.50

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

16

Storage for /

300

Storage for /nsm

3TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Recently I ran SOUP to update from v2.4.30 -> v2.4.50. There was an issue with Salt on one of my Forward nodes.

Following the advise from dougburks in this discussion here, I tried running so-saltstack-update.

Following which tonnes of errors. For example

1. salt

[master] so-checkin
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[INFO    ] Loading fresh modules for state activity
local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'data.*' in environment 'base' is not available on the salt master
----------
    Specified SLS 'global' in environment 'base' is not available on the salt master

2. so-status

[master] so-status

                         Security Onion Status                         
                         Container │ Status  │                 Details 
───────────────────────────────────┼─────────┼─────────────────────────
                 so-dockerregistry │ running │           Up 34 minutes 
                     so-elastalert │ missing │                         
                  so-elastic-fleet │ running │           Up 17 minutes 
 so-elastic-fleet-package-registry │ running │ Up 29 minutes (healthy) 
                  so-elasticsearch │ running │           Up 33 minutes 
                       so-idstools │ running │           Up 33 minutes 
                       so-influxdb │ running │ Up 34 minutes (healthy) 
                         so-kibana │ running │           Up 29 minutes 
                         so-kratos │ running │           Up 33 minutes 
                       so-logstash │ running │           Up 29 minutes 
                          so-mysql │ running │ Up 33 minutes (healthy) 
                          so-nginx │ missing │                         
                       so-playbook │ missing │                         
                          so-redis │ running │           Up 29 minutes 
                      so-sensoroni │ running │           Up 33 minutes 
                            so-soc │ missing │                         
                       so-soctopus │ running │           Up 18 minutes 
                       so-telegraf │ running │           Up 33 minutes 

 ❗ Check container logs and /opt/so/log for more details.

Further inspecting so-saltstack-update I see that it's trying to clone the securityonion Git repo. From what I can tell, this repo is configured for v2.3? Is this what is causing errors on my v2.4.50 setup?

 /usr/sbin/so-saltstack-update

clone_to_tmp() {

  # TODO Need to add a air gap option
  # Make a temp location for the files
  mkdir /tmp/sogh
  cd /tmp/sogh
  #git clone -b dev https://github.com/Security-Onion-Solutions/securityonion.git
  git clone https://github.com/Security-Onion-Solutions/securityonion.git
  cd /tmp

}

Please let me know if it's still safe to use so-saltstack-update on my v2.4 Onion setup. I am currently trying to roll back all my configurations from /nsm/backup/.

Any help is appreciate. Thanks.

edit: removed tagging user

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[DoughnutPeach]

Managed to restore most of the services. However, seems like so-curator is still missing.

so-status

                       Security Onion Status                        
                         Container │ Status  │              Details 
───────────────────────────────────┼─────────┼──────────────────────
                        so-curator │ missing │                      
                 so-dockerregistry │ running │           Up 3 hours 
                     so-elastalert │ running │           Up 3 hours 
                  so-elastic-fleet │ running │           Up 3 hours 
 so-elastic-fleet-package-registry │ running │ Up 3 hours (healthy) 
                  so-elasticsearch │ running │           Up 3 hours 
                       so-idstools │ running │           Up 3 hours 
                       so-influxdb │ running │ Up 3 hours (healthy) 
                         so-kibana │ running │           Up 3 hours 
                         so-kratos │ running │           Up 3 hours 
                       so-logstash │ running │           Up 3 hours 
                          so-mysql │ running │ Up 3 hours (healthy) 
                          so-nginx │ running │ Up 3 hours (healthy) 
                       so-playbook │ running │           Up 3 hours 
                          so-redis │ running │           Up 3 hours 
                      so-sensoroni │ running │           Up 3 hours 
                            so-soc │ running │           Up 3 hours 
                       so-soctopus │ running │           Up 3 hours 
                       so-telegraf │ running │           Up 3 hours 

 ❗ Check container logs and /opt/so/log for more details.

Ran so-docker-refresh, so-docker-prune, then salt-call state.highstate and so-curator-start. Got the following errors

.
.
----------
          ID: so-curator
    Function: docker_container.running
      Result: False
     Comment: Failed to pull master:5000/security-onion-solutions/so-curator:2.4.50: Error 404: manifest for master:5000/security-onion-solutions/so-curator:2.4.50 not found: manifest unknown: manifest unknown
     Started: 11:40:35.665166
    Duration: 73.521 ms
     Changes:   
.
.
Summary for local
-------------
Succeeded: 12 (changed=1)
Failed:     1
-------------
Total states run:     13

Would appreciate any advice to resolve this. Thanks!

P.S. some additional info

cat /etc/soversion 
2.4.50


salt-run --versions
Salt Version:
          Salt: 3006.3
 
Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: 5.0.2
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: oracle 9.3 
        locale: utf-8
       machine: x86_64
       release: 5.14.0-362.18.0.2.el9_3.x86_64
        system: Linux
       version: Oracle Linux Server 9.3 

[Bal33p]

I am unable to update salt also. I have rolled back 3 times due to this issue.
repo unavailable
wget https://repo.saltproject.io/salt/py3/redhat/9/$basearch/minor/3006.6/
--2024-03-05 14:28:17-- https://repo.saltproject.io/salt/py3/redhat/9//minor/3006.6/
Resolving repo.saltproject.io (repo.saltproject.io)... 18.154.227.73, 18.154.227.46, 18.154.227.74, ...
Connecting to repo.saltproject.io (repo.saltproject.io)|18.154.227.73|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-03-05 14:28:23 ERROR 404: Not Found.

[DoughnutPeach]

I assume you ran so-saltstack-update as well?

Do you still have backups of your salt and pillars prior to running the update? You can try to get them from /nsm/backup/ , then use rsync to copy all your old configs back to /opt/so/saltstack/.

[DoughnutPeach]

Just a note: so-saltstack-update will not completely reset your salt configurations cleanly.

I was having issues with SOC Administration on the GUI ref. Turns out the issue was that there were remnant v2.3 files in /opt/so/saltstack/default/salt/idh. The folders were skins and defaults.

I only realised this as I was manually compared the v2.4.50 Github repo here VS the master branch here.

So for future reference, avoid using so-saltstack-update if possible!! A lot of debugging and troubleshooting will be required to bring your environment back to standard state.

[petiepooo]

What did you specify as the branch when running so-saltstack-update?

There's a warning in the discussion you linked to that states you must specify a branch. Looking at the code, it defaults to master, which is the 2.3 line. You should probably be using 2.4.50, based on your answers when opening the discussion.

[DoughnutPeach]

Right, must've glossed over that. That seems to be the issue.

In case anyone else makes the same mistake

You can attempt to restore back your old salt configurations from /nsm/backup.

Just copy over those files into your /opt/so/saltstack/, replacing both local and default folders.

[DoughnutPeach]

Salt minions in distributed setup seems to not be able to replace v.2.3 files after running the wrong so-saltstack-update.

So I'm trying to rollback all my Salt configurations. However, I have a distributed setup, and it seems like Salt is not correctly updating the files.

i.e.) On one of my forward nodes, I attempt to do a sudo salt-call state.highstate. No matter how many times I run it, there will always be the same few file changes.

e.g.) /usr/sbin/soup on my node is still using the v2.3. version.

/usr/sbin/soup
.
.
preupgrade_changes() {
    # This function is to add any new pillar items if needed.
    echo "Checking to see if changes are needed."

    [[ "$INSTALLEDVERSION" == 2.3.0 || "$INSTALLEDVERSION" == 2.3.1 || "$INSTALLEDVERSION" == 2.3.2 || "$INSTALLEDVERSION" == 2.3.10 ]] && up_to_2.3.20
    [[ "$INSTALLEDVERSION" == 2.3.20 || "$INSTALLEDVERSION" == 2.3.21 ]] && up_to_2.3.30
    [[ "$INSTALLEDVERSION" == 2.3.30 || "$INSTALLEDVERSION" == 2.3.40 ]] && up_to_2.3.50
    [[ "$INSTALLEDVERSION" == 2.3.50 || "$INSTALLEDVERSION" == 2.3.51 || "$INSTALLEDVERSION" == 2.3.52 || "$INSTALLEDVERSION" == 2.3.60 || "$INSTALLEDVERSION" == 2.3.61 || "$INSTALLEDVERSION" == 2.3.70 ]] && up_to_2.3.80
    [[ "$INSTALLEDVERSION" == 2.3.80 ]] && up_to_2.3.90
    [[ "$INSTALLEDVERSION" == 2.3.90 || "$INSTALLEDVERSION" == 2.3.91 ]] && up_to_2.3.100
    [[ "$INSTALLEDVERSION" == 2.3.100 ]] && up_to_2.3.110
    [[ "$INSTALLEDVERSION" == 2.3.110 ]] && up_to_2.3.120
    [[ "$INSTALLEDVERSION" == 2.3.120 ]] && up_to_2.3.130
    [[ "$INSTALLEDVERSION" == 2.3.130 ]] && up_to_2.3.140
    [[ "$INSTALLEDVERSION" == 2.3.140 ]] && up_to_2.3.150
    [[ "$INSTALLEDVERSION" == 2.3.150 ]] && up_to_2.3.160
    [[ "$INSTALLEDVERSION" == 2.3.160 ]] && up_to_2.3.170
    [[ "$INSTALLEDVERSION" == 2.3.170 ]] && up_to_2.3.180
    [[ "$INSTALLEDVERSION" == 2.3.180 ]] && up_to_2.3.181
    [[ "$INSTALLEDVERSION" == 2.3.181 ]] && up_to_2.3.182
    [[ "$INSTALLEDVERSION" == 2.3.182 ]] && up_to_2.3.190
    [[ "$INSTALLEDVERSION" == 2.3.190 ]] && up_to_2.3.200
    [[ "$INSTALLEDVERSION" == 2.3.200 ]] && up_to_2.3.210
    [[ "$INSTALLEDVERSION" == 2.3.210 ]] && up_to_2.3.220
    [[ "$INSTALLEDVERSION" == 2.3.220 ]] && up_to_2.3.230
    [[ "$INSTALLEDVERSION" == 2.3.230 ]] && up_to_2.3.240
    [[ "$INSTALLEDVERSION" == 2.3.240 ]] && up_to_2.3.250
    [[ "$INSTALLEDVERSION" == 2.3.250 ]] && up_to_2.3.260
    [[ "$INSTALLEDVERSION" == 2.3.260 ]] && up_to_2.3.270
    [[ "$INSTALLEDVERSION" == 2.3.270 ]] && up_to_2.3.280
    [[ "$INSTALLEDVERSION" == 2.3.280 ]] && up_to_2.3.290
    
    true
}

But i clearly see Salt trying to replace it when I run salt-call state.highstate

diff:
                      --- 
                      +++ 
                       preupgrade_changes() {
                           # This function is to add any new pillar items if needed.
                           echo "Checking to see if changes are needed."
                       
                      -    [[ "$INSTALLEDVERSION" == 2.3.0 || "$INSTALLEDVERSION" == 2.3.1 || "$INSTALLEDVERSION" == 2.3.2 || "$INSTALLEDVERSION" == 2.3.10 ]] && up_to_2.3.20
                      -    [[ "$INSTALLEDVERSION" == 2.3.20 || "$INSTALLEDVERSION" == 2.3.21 ]] && up_to_2.3.30
                      -    [[ "$INSTALLEDVERSION" == 2.3.30 || "$INSTALLEDVERSION" == 2.3.40 ]] && up_to_2.3.50
                      -    [[ "$INSTALLEDVERSION" == 2.3.50 || "$INSTALLEDVERSION" == 2.3.51 || "$INSTALLEDVERSION" == 2.3.52 || "$INSTALLEDVERSION" == 2.3.60 || "$INSTALLEDVERSION" == 2.3.61 || "$INSTALLEDVERSION" == 2.3.70 ]] && up_to_2.3.80
                      -    [[ "$INSTALLEDVERSION" == 2.3.80 ]] && up_to_2.3.90
                      -    [[ "$INSTALLEDVERSION" == 2.3.90 || "$INSTALLEDVERSION" == 2.3.91 ]] && up_to_2.3.100
                      -    [[ "$INSTALLEDVERSION" == 2.3.100 ]] && up_to_2.3.110
                      -    [[ "$INSTALLEDVERSION" == 2.3.110 ]] && up_to_2.3.120
                      -    [[ "$INSTALLEDVERSION" == 2.3.120 ]] && up_to_2.3.130
                      -    [[ "$INSTALLEDVERSION" == 2.3.130 ]] && up_to_2.3.140
                      -    [[ "$INSTALLEDVERSION" == 2.3.140 ]] && up_to_2.3.150
                      -    [[ "$INSTALLEDVERSION" == 2.3.150 ]] && up_to_2.3.160
                      -    [[ "$INSTALLEDVERSION" == 2.3.160 ]] && up_to_2.3.170
                      -    [[ "$INSTALLEDVERSION" == 2.3.170 ]] && up_to_2.3.180
                      -    [[ "$INSTALLEDVERSION" == 2.3.180 ]] && up_to_2.3.181
                      -    [[ "$INSTALLEDVERSION" == 2.3.181 ]] && up_to_2.3.182
                      -    [[ "$INSTALLEDVERSION" == 2.3.182 ]] && up_to_2.3.190
                      -    [[ "$INSTALLEDVERSION" == 2.3.190 ]] && up_to_2.3.200
                      -    [[ "$INSTALLEDVERSION" == 2.3.200 ]] && up_to_2.3.210
                      -    [[ "$INSTALLEDVERSION" == 2.3.210 ]] && up_to_2.3.220
                      -    [[ "$INSTALLEDVERSION" == 2.3.220 ]] && up_to_2.3.230
                      -    [[ "$INSTALLEDVERSION" == 2.3.230 ]] && up_to_2.3.240
                      -    [[ "$INSTALLEDVERSION" == 2.3.240 ]] && up_to_2.3.250
                      -    [[ "$INSTALLEDVERSION" == 2.3.250 ]] && up_to_2.3.260
                      -    [[ "$INSTALLEDVERSION" == 2.3.260 ]] && up_to_2.3.270
                      -    [[ "$INSTALLEDVERSION" == 2.3.270 ]] && up_to_2.3.280
                      -    [[ "$INSTALLEDVERSION" == 2.3.280 ]] && up_to_2.3.290
                      -    
                      +    [[ "$INSTALLEDVERSION" == 2.4.2 ]] && up_to_2.4.3
                      +    [[ "$INSTALLEDVERSION" == 2.4.3 ]] && up_to_2.4.4
                      +    [[ "$INSTALLEDVERSION" == 2.4.4 ]] && up_to_2.4.5
                      +    [[ "$INSTALLEDVERSION" == 2.4.5 ]] && up_to_2.4.10
                      +    [[ "$INSTALLEDVERSION" == 2.4.10 ]] && up_to_2.4.20
                      +    [[ "$INSTALLEDVERSION" == 2.4.20 ]] && up_to_2.4.30
                      +    [[ "$INSTALLEDVERSION" == 2.4.30 ]] && up_to_2.4.40
                      +    [[ "$INSTALLEDVERSION" == 2.4.40 ]] && up_to_2.4.50
                           true
                       }

[TOoSmOotH]

#12567

This will be in 2.4.60 although this tool is really only used on the development side I have fixed it so it doesn't mess up any more existing 2.4 installs. In the meantime you can do so-saltstack-update 2.4/main to try and recover your grid.

[dougburks]

WARNING! For others that may come across this discussion, please note that so-saltstack-update is an undocumented command and is not used under normal circumstances. It is primarily a developer tool and should only be used if you are explicitly directed to do so.