Issues with Salt

[mciantar]

Hi,

I am having an issue with salt, but I cannot identify where the problem actually is.

Here are the logs I am getting:

---

`2023-03-07 02:30:43,592 [salt.pillar :1224][CRITICAL][428765] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
2023-03-07 02:30:44,707 [salt.client.mixins:442 ][INFO ][428758] Runner completed: 20230307023044681074
2023-03-07 02:30:44,821 [salt.client.mixins:442 ][INFO ][428770] Runner completed: 20230307023044792451
2023-03-07 02:30:45,935 [salt.client.mixins:442 ][INFO ][428758] Runner completed: 20230307023045305656
2023-03-07 02:30:45,936 [salt.utils.templates:274 ][ERROR ][428758] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "", line 27, in
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-03-07 02:30:45,936 [salt.pillar :900 ][CRITICAL][428758] Rendering SLS 'logstash.nodes' failed, render error:
Jinja variable list object has no element 0
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "", line 27, in
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/pillar/init.py", line 887, in render_pstate
state = compile_template(
File "/usr/lib/python3/dist-packages/salt/template.py", line 99, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/renderers/jinja.py", line 62, in render
tmp_data = salt.utils.templates.JINJA(
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-03-07 02:30:45,958 [salt.pillar :1224][CRITICAL][428758] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
2023-03-07 02:30:46,014 [salt.utils.job :85 ][INFO ][428771] Got return from so-node-manager_managersearch for job 20230307023046014300
2023-03-07 02:30:46,033 [salt.utils.event :905 ][ERROR ][428771] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-03-07 02:30:46,052 [salt.client.mixins:442 ][INFO ][428770] Runner completed: 20230307023045412284
2023-03-07 02:30:46,053 [salt.utils.templates:274 ][ERROR ][428770] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "", line 27, in
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-03-07 02:30:46,053 [salt.pillar :900 ][CRITICAL][428770] Rendering SLS 'logstash.nodes' failed, render error:
Jinja variable list object has no element 0
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "", line 27, in
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/pillar/init.py", line 887, in render_pstate
state = compile_template(
File "/usr/lib/python3/dist-packages/salt/template.py", line 99, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/renderers/jinja.py", line 62, in render
tmp_data = salt.utils.templates.JINJA(
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-03-07 02:30:46,064 [salt.pillar :1224][CRITICAL][428770] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
2023-03-07 02:30:46,147 [salt.utils.job :85 ][INFO ][428767] Got return from so-node-sensor-01_sensor for job 20230307023046146693
2023-03-07 02:30:46,147 [salt.utils.event :905 ][ERROR ][428767] Event iteration failed with exception: 'list' object has no attribute 'items'
`

Any pointers would be greatly appreciated.

I am running version 2.3.190, but cannot run soup to update to the latest version as is.

[andre-dessert]

Are you seeing anything more specific in the /opt/so/log/salt/master log?

[mciantar]

Hi @andre-dessert ,

The output I posted is from /opt/so/log/salt/master

From my basic understanding, it looks like /opt/so/saltstack/default/pillar/logstash/nodes.sls has something wrong, or data that it is parsing.

I am not sure when the problem started. The only thing that changed on the system recent is that a network card was replaced and the interface name changed. Could this be the issue?

Thanks!

[mciantar]

Is there anyone who can help me recover?

I cannot run soup at this stage.

[cm-ops]

Try the following and check the salt mine. salt-call mine.get '*' network.ip_addrs, salt-call pillar.get host:mainint, salt-call grains.get ip_interfaces. If the information for NIC and IP are correct, try salt-call mine.update.

[mciantar]

root@so-node-manager:~# salt-call mine.get '*' network.ip_addrs
local:
    ----------
    so-node-manager_managersearch:
    so-node-sensor-01_sensor:
        - <redacted>
root@so-node-manager:~# salt-call pillar.get host:mainint
local:
    ens18
root@so-node-manager:~# salt-call grains.get ip_interfaces
local:
    ----------
    docker0:
        - 172.17.0.1
    ens18:
        - <redacted>
        - <redacted>
        - fe80::3040:5cff:fea7:5938
    lo:
        - 127.0.0.1
    veth07b396c:
    veth432e921:
    veth4539de8:

Is the first output missing the IP address for the managersearch, or is that expected? The rest of the information looks fine. I called mine.update that returned True, but still have the same issue:

root@so-node-manager:~# so-checkin
[INFO    ] Loading fresh modules for state activity
local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering SLS 'logstash.nodes' failed. Please see master log for details.

[cm-ops]

There should be an IP address for the managersearch. Did you try and update the salt mine with salt-call mine.update? If not, try that and recheck salt-call mine.get '*' network.ip_addrs to verify it is not there.

[mciantar]

@cm-ops , thanks for your support - here is the output

root@so-node-manager:~# salt-call mine.update
local:
    True
root@so-node-manager:~# salt-call mine.get '*' network.ip_addrs
local:
    ----------
    so-node-manager_managersearch:
    so-node-sensor-01_sensor:
        - <reacted>

[cm-ops]

Would you validate this file is present? /etc/salt/minion.d/mine_functions.conf

[mciantar]

Thanks @cm-ops ! That was it.

This server required the interface to be changed. The file you pointed me to still had the old interface name. I updated it and ran salt-call mine.update and things are now working!

Thanks a lot!