2.4.10 soup caused permanent highstate

[kennyrogersjr]

SO 2.4.10 soft-broke after running sudo soup. We originally wanted to restart the system due to Kibana web services not responding, however, we were advised to run soup prior to that. Upon soup finishing, init 6 and reboot wouldn't restart the system and ended up using systemctl reboot -i.

Receiving the below when running sudo salt-call state.highstate. We even let the server sit for a full day and received the following response from sudo so-status, "System appears to be starting. No highstate has completed since the system was restarted."

[redacted@redacted ~]$ sudo salt-call state.highstate
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 34, in top-level template code
jinja2.exceptions.UndefinedError: the template 'salt/map.jinja' (imported on line 2) does not export the requested name 'UPGRADECOMMAND'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable the template 'salt/map.jinja' (imported on line 2) does not export the requested name 'UPGRADECOMMAND'; line 34

---
[...]
  cmd.run:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c '{{ UPGRADECOMMAND }}' &    <======================

{% endif %}

{% if INSTALLEDSALTVERSION|string == SALTVERSION|string %}

[...]
---
[CRITICAL] Rendering SLS 'base:salt.minion' failed: Jinja variable the template 'salt/map.jinja' (imported on line 2) does not export the requested name 'UPGRADECOMMAND'; line 34

---
[...]
  cmd.run:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c '{{ UPGRADECOMMAND }}' &    <======================

{% endif %}

{% if INSTALLEDSALTVERSION|string == SALTVERSION|string %}

[...]
---
local:
    Data failed to compile:
----------
    Rendering SLS 'base:salt.minion' failed: Jinja variable the template 'salt/map.jinja' (imported on line 2) does not export the requested name 'UPGRADECOMMAND'; line 34

---
[...]
  cmd.run:
    - name: |
        exec 0>&- # close stdin
        exec 1>&- # close stdout
        exec 2>&- # close stderr
        nohup /bin/sh -c '{{ UPGRADECOMMAND }}' &    <======================

{% endif %}

{% if INSTALLEDSALTVERSION|string == SALTVERSION|string %}

[...]
---

The /opt/so/log/salt/master log doesn't show anything interesting. However, we're noticing that elastic is showing 85% full errors.

I even gave soup another shot and now it wants to upgrade from 2.4.10 to 2.4.60, but refuses due to the highstate issue.

Anybody have any clues to look for any root or underlying issues? Better yet, anyone know how to get back to functional?

[reyesj2]

Is this an ISO install? If not what OS are you using?

What version of salt is installed?
dnf info salt*

[kennyrogersjr]

3006.6

Please note, I yummed it up yesterday with a salt reinstall and restored from /nsm. We're still troubleshooting.

[kennyrogersjr]

So we're standing up a new instance with 2.4.60 and porting the configs over. We don't know what the previous admin changed and where the hiccup is post-soup. The SO docs even say it's easier to just stand up a new instance versus troubleshooting. Closing it out with this.

[kennyrogersjr]

Closed.