Sensors failing to start any services "Data failed to compile"

[jtrodriguez]

Version

2.4.80

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

256GB

Storage for /

256GB

Storage for /nsm

64TB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I have 3 sensors and none are booting properly after the manager had been powered down for 12 hours.

On so-status I receive this message

⌛ System appears to be starting. No highstate has completed since the system was restarted.

They all have this error on "so-checkin"

[INFO ] Executing command 'yum' in directory '/root'
[INFO ] Executing command 'yum' in directory '/root'
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
self.environment.handle_exception()
File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
raise rewrite_traceback_stack(source=source)
File "", line 119, in top-level template code
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 247, in str
output.append(f"{key!r}: {value!s}")
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 247, in str
output.append(f"{key!r}: {value!s}")
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/jinja.py", line 247, in str
output.append(f"{key!r}: {value!s}")
jinja2.exceptions.UndefinedError: list object has no element 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable list object has no element 0; line 119

---

[...]
suriconfig:
file.managed:
- name: /opt/so/conf/suricata/suricata.yaml
- source: salt://suricata/files/suricata.yaml.jinja
- context:
suricata_config: {{ SURICATAMERGED.config }} <======================
- user: 940
- group: 940
- template: jinja

surithresholding:
[...]

[CRITICAL] Rendering SLS 'base:suricata.config' failed: Jinja variable list object has no element 0; line 119

---

[...]
suriconfig:
file.managed:
- name: /opt/so/conf/suricata/suricata.yaml
- source: salt://suricata/files/suricata.yaml.jinja
- context:
suricata_config: {{ SURICATAMERGED.config }} <======================
- user: 940
- group: 940
- template: jinja

surithresholding:
[...]

[INFO ] Executing command /bin/systemctl in directory '/root'
local:
Data failed to compile:

Rendering SLS 'base:suricata.config' failed: Jinja variable list object has no element 0; line 119

---

[...]
suriconfig:
file.managed:
- name: /opt/so/conf/suricata/suricata.yaml
- source: salt://suricata/files/suricata.yaml.jinja
- context:
suricata_config: {{ SURICATAMERGED.config }} <======================
- user: 940
- group: 940
- template: jinja

surithresholding:
[...]

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Can you show me what sudo salt-call pillar.get suricata returns?

[jtrodriguez]

Manager Server:
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
local:

One of the sensors that is missing:
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
local:

config:

af-packet:

threads:
31
vars:

address-groups:

AD_SERVERS:

• 10.1.28.***
• 10.1.28.***
• 10.1.24.***
• 10.250.10.***
• 10.250.10.***
• 10.250.10.***
  DNS_SERVERS:
• 10.1.24.***
• 10.1.24.***
• 10.250.0.***
• 10.250.0.***
• 10.250.10.***
• 10.250.10.***
  SMTP_ACCESS:
  SNMP_ACCESS:
• 10.1.27.***
• 10.1.28.***
• 10.250.10.***
• 10.1.28.***
• 10.1.27.***
  enabled:
  True
  pcap:

---

maxsize:
17577

[cm-ops]

Is there a config in SMTP_ACCESS:? It is showing up but there is nothing configured.

[jtrodriguez]

There isn't anything for that variable. I created it and never utilized it.

[cm-ops]

Try removing that entry from the pillar file (/opt/so/saltstack/local/pillar/suricata/soc_suricata.sls) and apply the Suricata state.

[jtrodriguez]

Not sure why that would fix it but it did. Thank you!