System appears to be starting. No highstate has completed since the system was restarted.

[larry1788]

Have you ever encountered this problem? After the SO system was upgraded to 2.4.40 a few days ago, I received a system notification today that I needed to reboot. I followed the system instructions to reboot. In the end, it failed to operate normally.

[soc ~]$ sudo so-status
[sudo] password for soc:
⌛ System appears to be starting. No highstate has completed since the system was restarted.

[soc@solab-m ~]$ sudo so-checkin
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory 'grains/' for environment 'base'
[ERROR ] Exception in callback functools.partial(<function wrap..null wrapper at 0x7f4c40ef88b0>, <salt.ext.tornado.concurrent.Future object at 0x7f4c 40f7d510>)
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 603, in _send_recv
recv = yield self.socket.recv()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py ", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
zmq.eventloop.future.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 605, in _send_recv
future.set_exception(exc)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 294, in set_exception
self.set_exc_info(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 319, in set_exc_info
self._set_done()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 333, in _set_done
for cb in self._callbacks:
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 606, in run_callback
ret = callback()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 628, in _discard_future_result
future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py ", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 612, in _send_recv
future.set_exception(exc)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 294, in set_exception
self.set_exc_info(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 319, in set_exc_info
self._set_done()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 333, in _set_done
for cb in self.callbacks:
TypeError: 'NoneType' object is not iterable
[ERROR ] Exception in callback functools.partial(<function wrap..null wrapper at 0x7f4c40ef9000>, <salt.ext.tornado.concurrent.Future object at 0x7f4c 40ef4a30>)
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 601, in _send_recv
yield self.socket.send(message)
AttributeError: 'NoneType' object has no attribute 'send'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 606, in run_callback
ret = callback()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 628, in _discard_future_result
future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py ", line 1070, in run
yielded = self.gen.send(value)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 612, in _send_recv
future.set_exception(exc)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 294, in set_exception
self.set_exc_info(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 319, in set_exc_info
self._set_done()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 333, in _set_done
for cb in self.callbacks:
TypeError: 'NoneType' object is not iterable
[ERROR ] Exception in callback functools.partial(<function wrap..null wrapper at 0x7f4c40ef97e0>, <salt.ext.tornado.concurrent.Future object at 0x7f4c 40ef53c0>)
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 601, in _send_recv
yield self.socket.send(message)
AttributeError: 'NoneType' object has no attribute 'send'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 606, in run_callback
ret = callback()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop .py", line 628, in _discard_future_result
future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py ", line 1070, in run
yielded = self.gen.send(value)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.p y", line 612, in _send_recv
future.set_exception(exc)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 294, in set_exception
self.set_exc_info(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 319, in set_exc_info
self._set_done()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concur rent.py", line 333, in _set_done
for cb in self._callbacks:
TypeError: 'NoneType' object is not iterable
Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout error

May I ask the experts how to deal with this?

Thank ！

I really thought about the worst case scenario and reinstalled the system.

[cm-ops]

Did you check your soup log for any issues with upgrade?

[larry1788]

[soc@solab-m ~]$ sudo systemctl status salt-master
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; preset: disabled)
Active: active (running) since Thu 2024-02-01 02:14:43 UTC; 4min 38s ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltproject.io/en/latest/contents.html
Main PID: 88814 (/opt/saltstack/)
Tasks: 41 (limit: 99158)
Memory: 531.5M
CPU: 4min 45.333s
CGroup: /system.slice/salt-master.service
├─ 88814 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess"
├─ 89205 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon"
├─ 89206 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher"
├─ 89213 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Engine(salt.loaded.ext.engines.checkmi>
├─ 89214 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance"
├─ 89215 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager"
├─ 89216 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue"
├─ 89217 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0"
├─ 89218 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1"
├─ 89219 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate"
├─ 89220 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2"
├─ 89228 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3"
├─ 89230 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4"

[larry1788]

/opt/so/log/salt/master

2024-02-01 00:14:18,502 [salt.client.mixins:441 ][INFO ][22909] Runner completed: 20240201001418454736
2024-02-01 00:14:19,534 [salt.client.mixins:441 ][INFO ][22909] Runner completed: 20240201001418942015
2024-02-01 00:14:20,044 [salt.client.mixins:441 ][INFO ][22909] Runner completed: 20240201001420011351
2024-02-01 00:14:21,182 [salt.client.mixins:441 ][INFO ][22909] Runner completed: 20240201001420523183
2024-02-01 00:14:29,668 [salt.client.mixins:441 ][INFO ][22905] Runner completed: 20240201001429610227
2024-02-01 00:14:30,937 [salt.client.mixins:441 ][INFO ][22905] Runner completed: 20240201001430302012
2024-02-01 00:14:31,550 [salt.client.mixins:441 ][INFO ][22905] Runner completed: 20240201001431514134
2024-02-01 00:14:32,964 [salt.client.mixins:441 ][INFO ][22905] Runner completed: 20240201001432053168
2024-02-01 00:14:34,636 [salt.utils.job :86 ][INFO ][22898] Got return from solab-s_searchnode for job 202402010
01434635329
2024-02-01 00:14:34,637 [salt.utils.event :912 ][ERROR ][22898] Event iteration failed with exception: 'list' object
has no attribute 'items'
2024-02-01 00:14:39,136 [salt.utils.job :86 ][INFO ][22897] Got return from solab-s_searchnode for job 202402010
01439135033
2024-02-01 00:14:46,126 [salt.utils.job :86 ][INFO ][22898] Got return from solab-m_manager for job 202402010014
46125288
2024-02-01 00:14:46,126 [salt.utils.event :912 ][ERROR ][22898] Event iteration failed with exception: 'list' object
has no attribute 'items'

2024-02-01 00:16:56,154 [salt.transport.zeromq:359 ][INFO ][22895] MWorkerQueue under PID 22895 is closing
2024-02-01 00:17:01,158 [salt.utils.process:744 ][INFO ][22254] Some processes failed to respect the KILL signal: P
rocess: (Pid: 22889)
2024-02-01 00:17:01,216 [salt.utils.process:751 ][INFO ][22254] kill_children retries left: 3
2024-02-01 00:17:01,218 [salt.utils.parsers:1066][WARNING ][22254] Master received a SIGTERM. Exiting.
2024-02-01 00:17:01,218 [salt.cli.daemons :99 ][INFO ][22254] The Salt Master is shut down

2024-02-01 02:21:01,471 [salt.client.mixins:441 ][INFO ][89217] Runner completed: 20240201022100899188
2024-02-01 02:21:06,239 [salt.utils.job :86 ][INFO ][89220] Got return from solab-m_manager for job 202402010221
06238283
2024-02-01 02:21:06,240 [salt.utils.event :912 ][ERROR ][89220] Event iteration failed with exception: 'list' object
has no attribute 'items'
2024-02-01 02:21:10,312 [salt.client.mixins:441 ][INFO ][89217] Runner completed: 20240201022110100049

[cm-ops]

What does the following show? cat /etc/soversion, salt-run --versions

[Fenr1r-g]

Not OP, but have the same issue upon install.

cat /etc/soversion = 2.4.50

salt-run --versions = command not found.

[cm-ops]

What does dnf info salt* return?

[kirangowda29]

Facing same issue in 2.4.40. any solution?

[Bal33p]

Experiencing same issue trying to patch to .50 salt wont install

[Bal33p]

wget https://repo.saltproject.io/salt/py3/redhat/9/$basearch/minor/3006.6/
--2024-03-05 14:28:17-- https://repo.saltproject.io/salt/py3/redhat/9//minor/3006.6/
Resolving repo.saltproject.io (repo.saltproject.io)... 18.154.227.73, 18.154.227.46, 18.154.227.74, ...
Connecting to repo.saltproject.io (repo.saltproject.io)|18.154.227.73|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-03-05 14:28:23 ERROR 404: Not Found.

[Fenr1r-g]

Looks like the script call for $basearch is not working correctly. Not sure what the variable is supposed to be...I'm guessing the result of parsing something like "uname -a" to determine what arch it needs to select...

https://repo.saltproject.io/salt/py3/redhat/9/**$basearch**/minor/3006.6/

When looking at the base of the repo (https://repo.saltproject.io/salt/py3/redhat/9/) here's what I see.

[KineticByte]

Not OP, but have the same issue after upgrade. Get the following error when run $sudo so-checkin.

local:
    Data failed to compile:
----------
    Rendering SLS 'base:salt.minion' failed: Jinja variable the template 'salt/map.jinja' (imported on line 2) does not export the requested name 'UPGRADECOMMAND'; line 34

---
[...]
  cmd.run:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c '{{ UPGRADECOMMAND }}' &    <======================

{% endif %}

{% if INSTALLEDSALTVERSION|string == SALTVERSION|string %}

[...]