salt upgrade process bug?

[palichx]

Hello,
It was so long upgrade(2.91 - 2.100) for my setup :) :

1. soup was faild with error:

Updated:
Verifying : salt-master-3003-1.el7.noarch 4/6
Verifying : salt-3003-1.el7.noarch 5/6
Verifying : salt-minion-3003-1.el7.noarch 6/6

Updated:
salt-master.noarch 0:3004-1.el7 salt-minion.noarch 0:3004-1.el7

Dependency Updated:
salt.noarch 0:3004-1.el7

Complete!
^[[1;32m * INFO^[[0m: Running install_centos_stable_post()
^[[1;32m * INFO^[[0m: Running install_centos_check_services()
^[[1;32m * INFO^[[0m: Salt installed! (0)
Applying yum versionlock for Salt.

Loaded plugins: versionlock
Adding versionlock on: 0:salt-master-3004-1.el7
Adding versionlock on: 0:salt-minion-3004-1.el7
Adding versionlock on: 0:salt-3004-1.el7
versionlock added: 3
Checking if Salt was upgraded.

Salt upgrade success.

Checking to see if changes are needed.
Updating to Security Onion 2.3.100
Detecting if /nsm/wazuh/etc/ossec.conf needs corrected...
/nsm/wazuh/etc/ossec.conf does not have an XML header, so no changes are necessary.
Removing /opt/so/state files for patched Salt InfluxDB module and state. This is due to Salt being upgraded and needing to patch the files again.
removed ‘/opt/so/state/influxdb_continuous_query.py.patched’
removed ‘/opt/so/state/influxdb_retention_policy.py.patched’
removed ‘/opt/so/state/influxdbmod.py.patched’
Adding receiver hostgroup with so-firewall
receiver hostgroup added
Adding receiver to assigned_hostgroups.local.map.yaml
Soup failed with error 2: No such file or directory

There is no file assigned_hostgroups.local.map.yaml in /opt/so/saltstack/local/salt/firewall/. (fixed)

2. Next run soup finished with other error:

Updating dockers to 2.3.100.
Soup failed with error 1: Unhandled error
Unhandled error occured, please check /root/soup.log for details.

Starting crond service at 11:11:58.926286
Successfully started crond.

Ensuring highstate is enabled.
local:
----------
msg:
Info: highstate state already enabled.
res:
True

3. after reboot manager status of all containers was MISSING.

4. manually so-restart any of them finished with error:

so-elasticsearch-restart
so-elasticsearch
[ERROR ] Failure!
local:
ID: elasticsearch_state_not_allowed
Function: test.fail_without_changes
Result: False
Comment: Failure!
Started: 09:55:43.139444
Duration: 2.679 ms

4. I found my problem in /opt/so/saltstack/default/salt/salt/minion.defaults.yaml. This file contains incorrect salt version

salt:
minion:
version: 3003

manager#salt-minion --version
salt-minion 3004

As result condition {% if grains.saltversion | string == saltversion | string %} in file /opt/so/saltstack/default/salt/allowed_states.map.jinja was FALSE.
Manually changing version solve problem.

---

As i understand in soup process only salt package was updated. Sls file was not updated.

May be need to upgrade salt package and salt sls in independent process?

[dougburks]

I don't believe we've seen this issue in any of our internal tests and I don't believe anybody else has reported it either.

Before the upgrade, had you made any modifications to your system?

[palichx]

Hi, thanks for answer.
yes i have some modification in "/opt/so/saltstack/local/":
elasticsearch pipilnes/logstash/filebeat/zeek. i didn't touch salt.sls
I deleted /opt/so/saltstack/local/salt/firewall/assigned_hostgroups.local.map.yaml for some my test and foget about it (my fault). But i think its ok if i have default file in /opt/so/saltstack/default/salt/firewall/.

I will try to repeat my case in a day.

[palichx]

yep, case is repeateble.
It's fresh standalone installations 2.3.91:

1. I remove file /opt/so/saltstack/local/salt/firewall/assigned_hostgroups.local.map.yaml and run soup.
2. soup finished with error:
   
3. salt version is 3004
   
4. salt sls contains previos version
   

[dougburks]

I deleted /opt/so/saltstack/local/salt/firewall/assigned_hostgroups.local.map.yaml for some my test and foget about it (my fault).

We don't recommend deleting /opt/so/saltstack/local/salt/firewall/assigned_hostgroups.local.map.yaml.

But i think its ok if i have default file in /opt/so/saltstack/default/salt/firewall/.

There is a file called /opt/so/saltstack/default/salt/firewall/assigned_hostgroups.map.yaml but we still don't recommend deleting /opt/so/saltstack/local/salt/firewall/assigned_hostgroups.local.map.yaml.