Manager containers stuck at WAIT_START

[leskin]

Upon reboot, most of my manager node containers are stuck at WAIT_START, as shown by the so-status output below.

Checking Docker status

    Docker ---------------------------- [ OK ]

Checking container statuses

    so-aptcacherng -------------------- [ OK ]
    so-dockerregistry ----------------- [ OK ]
    so-elastalert ------------- [ WAIT_START ]
    so-elasticsearch ---------- [ WAIT_START ]
    so-filebeat --------------- [ WAIT_START ]
    so-fleet ------------------ [ WAIT_START ]
    so-grafana ---------------- [ WAIT_START ]
    so-idstools --------------- [ WAIT_START ]
    so-influxdb --------------- [ WAIT_START ]
    so-kibana ----------------- [ WAIT_START ]
    so-kratos ------------------------- [ OK ]
    so-logstash --------------- [ WAIT_START ]
    so-mysql ------------------ [ WAIT_START ]
    so-nginx ------------------ [ WAIT_START ]
    so-playbook --------------- [ WAIT_START ]
    so-redis ------------------ [ WAIT_START ]
    so-sensoroni -------------- [ WAIT_START ]
    so-soc -------------------- [ WAIT_START ]
    so-soctopus --------------- [ WAIT_START ]
    so-telegraf --------------- [ WAIT_START ]
    so-wazuh ------------------ [ WAIT_START ]

System details:

Security Onion Version: 2.3.190
Distributed deployment with manager node running in a virtual machine, one search node and one sensor node, each in dedicated hardware.
The system was installed from the ISO image in CentOS 7

What I've Tried:

The command

sudo salt-call state.highstate

fails with the following errors:

2023-01-18 15:40:18,598 [salt.transport.mixins.auth:212 ][INFO    ][1853] Authentication request from so-manager_manager
2023-01-18 15:40:18,607 [salt.transport.mixins.auth:488 ][INFO    ][1853] Authentication accepted from so-manager_manager
2023-01-18 15:40:21,233 [salt.client.mixins:442 ][INFO    ][1859] Runner completed: 20230118154021083033
2023-01-18 15:40:24,107 [salt.client.mixins:442 ][INFO    ][1859] Runner completed: 20230118154022807579
2023-01-18 15:40:24,115 [salt.utils.templates:274 ][ERROR   ][1859] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 29, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-01-18 15:40:24,117 [salt.pillar      :900 ][CRITICAL][1859] Rendering SLS 'logstash.nodes' failed, render error:
Jinja variable list object has no element 0
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 29, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 896, in render_pstate
    **defaults
  File "/usr/lib/python3.6/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 75, in render
    **kws
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-01-18 15:40:24,237 [salt.pillar      :1224][CRITICAL][1859] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
2023-01-18 15:40:25,181 [salt.utils.job   :85  ][INFO    ][1856] Got return from so-manager_manager for job 20230118154025177401
2023-01-18 15:40:25,186 [salt.utils.event :908 ][ERROR   ][1856] Event iteration failed with exception: 'list' object has no attribute 'items'

The error log leads me to believe there is a configuration error somewhere, and these symptoms are similar to discussion #9519, but I have not made any physical hardware changes or modified any configuration files (but it has been a while since I rebooted the manager node). I followed the recommendations made in discussion #9519 but unfortunately that did not solve the problem.

When I ran the command

sudo salt-call mine.get '*' network.ip_addrs

ip addresses are listed for the manager and sensor node, but no ip address is listed for the search node, leading me to think that may be related to the problem. When I ran the commands

salt-call pillar.get host:mainint 
salt-call grains.get ip_interfaces

the manager node interface information returned was correct. I am able to ping both the search node and the sensor node from the manager node, so there does not seem to be a network connectivity issue.

I've done some digging to look at the lines listed in the tracebacks but have not had much success there yet. I'm far from a salt guru, so any help diagnosing the issue would be much appreciated.

Thanks,

//Leo

[cm-ops]

Have you tried updating the salt mine? If salt \* test.ping works and returns your nodes, try salt-call mine.update, then run salt \* mine.update and recheck the mine.

[leskin]

The salt \* test.ping failed for both the sensor and search nodes. Both of those machines are also stuck in a WAIT_START state because I tried rebooting them after the manager got stuck. They appear to be stuck in the WAIT_START state because the manager is not responding to them during their startup (I didn't realize that could happen). They are not responding to the salt ping but those machines do respond to a normal ping from the manager.

#salt \* test.ping
so-manager_manager:
    True
so-forward1_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20230119223210729859
so-search1_searchnode:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20230119223210729859
ERROR: Minions returned with non-zero exit code

[cm-ops]

Would you check the salt master service - systemctl status salt-master and the master log /opt/so/log/salt/master for errors.

[leskin]

systemctl status salt-master returns

● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-01-19 22:17:26 UTC; 3 days ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 1064 (salt-master)
    Tasks: 54
   Memory: 712.3M
   CGroup: /system.slice/salt-master.service
           ├─ 1064 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1838 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1846 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1849 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1850 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1851 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1852 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1853 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1860 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1861 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1862 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1863 /usr/bin/python3 /usr/bin/salt-master
           ├─ 1865 /usr/bin/python3 /usr/bin/salt-master
           └─23350 /usr/bin/python3 /usr/bin/salt-master

Jan 23 19:46:11 so-manager salt-master[1064]: File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 75, in render
Jan 23 19:46:11 so-manager salt-master[1064]: **kws
Jan 23 19:46:11 so-manager salt-master[1064]: File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
Jan 23 19:46:11 so-manager salt-master[1064]: output = render_str(tmplstr, context, tmplpath)
Jan 23 19:46:11 so-manager salt-master[1064]: File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
Jan 23 19:46:11 so-manager salt-master[1064]: raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
Jan 23 19:46:11 so-manager salt-master[1064]: salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
Jan 23 19:46:11 so-manager salt-master[1064]: [CRITICAL] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
Jan 23 19:46:12 so-manager salt-master[1064]: [INFO    ] Got return from so-manager_manager for job 20230123194612014948
Jan 23 19:46:12 so-manager salt-master[1064]: [ERROR   ] Event iteration failed with exception: 'list' object has no attribute 'items'

and /opt/so/log/salt/master has the following:

2023-01-23 19:41:07,231 [salt.utils.templates:274 ][ERROR   ][1862] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 29, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-01-23 19:41:07,233 [salt.pillar      :900 ][CRITICAL][1862] Rendering SLS 'logstash.nodes' failed, render error:
Jinja variable list object has no element 0
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 29, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 896, in render_pstate
    **defaults
  File "/usr/lib/python3.6/site-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/renderers/jinja.py", line 75, in render
    **kws
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0
2023-01-23 19:41:07,352 [salt.pillar      :1224][CRITICAL][1862] Pillar render error: Rendering SLS 'logstash.nodes' failed. Please see master log for details.
2023-01-23 19:41:07,804 [salt.utils.job   :85  ][INFO    ][1865] Got return from so-manager_manager for job 20230123194107799672
2023-01-23 19:41:07,808 [salt.utils.event :908 ][ERROR   ][1865] Event iteration failed with exception: 'list' object has no attribute 'items'

[cm-ops]

On the searchnode, please check /etc/salt/minion.d/mine_functions.conf to verify the interface name is defined under network.ip_addrs. Also, please check the following: salt-call pillar.get host:mainint and salt-call grains.get ip_interfaces. You can also check the salt ports with nc -vz $MANAGER_IP 4505 (and 4506). Last, verify the time is in sync.

[cm-ops]

On the manager, if you check so-firewall includedhosts search_node and so-firewall includedhosts minion is the IP of the searchnode in both?

[leskin]

Yes, both commands list the IP of the searchnode, as well as the loopback address. Is it correct to have the manager loopback IP included in the search_node hostgroup? Seems odd. The sensor hostgroup lists the correct sensor IP along with the loopback address as well.

[cm-ops]

That is correct. So the manager has the appropriate FW settings. Is there an appliance between the nodes?

[leskin]

The manager node is a VM in a Proxmox host and the nodes are in adjacent ports in a router. There is an internal firewall as well so that is a potential culprit but no firewall rules have changed from when it was working. Perhaps the Proxmox security update somehow has fouled things up (in a non-obvious way)? I could recreate a new manager node VM from the ISO but I've held off doing that from concern that might confuse the authentication with the search and sensor nodes and create additional confusion.

[cm-ops]

It looks like all the pieces are there for communication from the minion to manager via salt. I would double check Proxmox FW settings to verify, there is nothing preventing the traffic on those ports.