Salt-master not available after upgrade 2.3.91 -> 2.3.100

[krestian83]

Hello,

As the title says the salt-master has stopped functioning properly after I tried to upgrade my distributed deployment. The salt-master service is running, but with a couple of errors according to /opt/so/logs/salt/master:

2022-02-21 09:55:51,225 [tornado.general :435 ][ERROR ][2768] Uncaught exception in ZMQStream callback
2022-02-21 11:20:56,698 [salt.utils.event :908 ][ERROR ][1976] Event iteration failed with exception: 'list' object has no attribute 'items'
2022-02-21 10:26:33,541 [salt.master :1611][ERROR ][1971] Received minion error from [ids-XXX-01_sensor]: The minion function caused an exception

Also, only the master has been upgraded to 2.3.100 while all nodes are still 2.3.91. About half of the containers fail to start on the master.

I have tried running soup again and it fails with this error:

Preparing soup at Fri Feb 18 08:00:43 UTC 2022

Checking to see if this is a manager.

Please run this command on the manager; the manager controls the grid.
Soup failed with error 1: Unhandled error
Unhandled error occured, please check /root/soup.log for details.

Starting service at 08:03:54.872938
Failed to start .

Ensuring highstate is enabled.
local:
----------
msg:
Info: highstate state already enabled.
res:
True

I am attaching the original soup.log

Please let me know of there is some additional info that could be relevant that is missing.

Br,

Christian

[krestian83]

Anyone?

[palichx]

Hi,

About half of the containers fail to start on the master.

what error do you get when you restart container?

[krestian83]

Hello,

When I restart for instance elasticsearch I get:

Restarting elasticsearch...

This could take a while if another Salt job is running.
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
so-elasticsearch
so-elasticsearch
local:
    Data failed to compile:
----------
    No matching sls found for 'elasticsearch' in env 'base'