Problem adding forward nodes, giving: Rendering SLS 'base:ssl' failed: Jinja variable list object has no element 0; line 48

[TheAOWizard]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

100

Storage for /nsm

500

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

We have just made a clean setup from scratch where we wiped the old setup and made a totally fresh install to upgrade to 2.4.

Our setup consist of:

• 1 x Manager Node (so-manager-v01)
• 3 x Search nodes (so-search-v01, so-search-v02, so-search-v03)
• 2 x Receiver nodes (so-receiver-v01, so-receiver-v02)
• 3 x Forward nodes (so-sensor-f01, so-sensor-f02, so-sensor-f03)

We started setting up the manager node, then the search nodes and the receiver nodes and everything was running fine, we then added the first forward node (so-sensor-f01) and we started getting data from our span ports.

The problem started when we tried to add the second (so-sensor-f02) and third (so-sensor-f03) forward node. The installation went fine, they popped up in Grid Members and we accepted them.

The first indication we saw that there was a problem was that so-sensor-f02 never showed up in the grid (so-sensor-f03 is in the grid overview and is showing healthy)

so-sensor-f02

To start the debugging we looked at /opt/so/log/ on so-sensor-f02, the only entry here is salt/minion that gives us this error:

2024-02-02 09:10:06,174 [salt.utils.templates:231 ][ERROR   ][2450501] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 48, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0; line 48

---
[...]

# Trust the CA
trusttheca:
  x509.pem_managed:
    - name: /etc/pki/tls/certs/intca.crt
    - text:  {{ trusttheca_text }}    <======================

{% if GLOBALS.os_family == 'Debian' %}
symlinkca:
  file.symlink:
    - target: /etc/pki/tls/certs/intca.crt
[...]
---
2024-02-02 09:10:06,174 [salt.state       :4273][CRITICAL][2450501] Rendering SLS 'base:ssl' failed: Jinja variable list object has no element 0; line 48

---
[...]

# Trust the CA
trusttheca:
  x509.pem_managed:
    - name: /etc/pki/tls/certs/intca.crt
    - text:  {{ trusttheca_text }}    <======================

{% if GLOBALS.os_family == 'Debian' %}
symlinkca:
  file.symlink:
    - target: /etc/pki/tls/certs/intca.crt
[...]
---

The command sudo salt-call state.highstate looks fine on the manager but on so-sensor-f02 it gives us the same error as the log above, but I'm not sure I saw the same error after initial install.

Looking at the discussion forum I found this old discussion 2.3.91 Upgrade Failing to 2.3.100 - Rendering SLS 'base:ssl' failed: Jinja variable list object has no element 0 #7054 with the same error message, but their problem is a capital letter in the host name, that's not our problem since all our hostnames are entirely lowercase.
The outputs from sudo salt-call pillar.get host:mainint, sudo salt-call grains.get ip_interfaces and sudo salt-call mine.get so-manager-v01_manager x509.get_pem_entries from so-sensor-f02 all looks normal.

so-sensor-f03

On so-sensor-f03 running the command sudo salt-call state.highstate gives us this error:

----------
          ID: run_installer
    Function: cmd.script
        Name: salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Command 'salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64' run"
              Command 'salt://elasticfleet/files/so_agent-installers/so-elastic-agent_linux_amd64' run
     Started: 09:50:11.885876
    Duration: 38125.991 ms
     Changes:
              ----------
              pid:
                  409377
              retcode:
                  1
              stderr:
              stdout:

                  Installation initiated, view install log for further details.
----------

Additional info

Neither so-sensor-f02 nor so-sensor-f03 are showing up in the elastic fleet

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Did the intermediate ca get created? /etc/pki/tls/certs/intca.crt

[m0duspwnens]

Is this issue still occuring? If not, it was likely resolved by the mine repopulating.

If so, from the sensor, when you run salt-call mine.get so-manager-v01_manager x509.get_pem_entries does it output your manager name followed by the ca? If you run salt-call grains.get master and compare that to the output from the mine.get do the manager names match? This would include case but exclude the _manager or _managersearch.

[KimkimCyber]

Can you tell me in what files the name of the manager should be changed on the search node?
I only found 3 instances on the search node, where I replaced the name with the correct one:
/etc/hosts
/etc/salt/minion
/etc/yum.repos.d/securityonion.repo

and then ran sudo salt '*' state.highstate

However, logstash is still referring to the incorrect one on the search node.

[TheAOWizard]

We ended up reinstalling the node, and that worked.