No such container: so-filebeat

[rabeyer]

During upgrade from 2.3.130 to 2.3.140, I encountered the issue referenced in discussion #8324 and implemented the fix suggested. During running that fix, I received the error referenced in discussion #8340 and, as suggested, removed the custom filebeat rule to enable Fortigate log parsing in /opt/so/saltstack/local/pillar/minions/so_standalone.sls and tried soup again.

I am now in the situation where most containers are in WAIT_START and attempting to manually start filebeat results in the error below.

local:
Error response from daemon: No such container: so-filebeat
Error: No such container: so-filebeat
[ERROR ] Rendering exception occurred
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "", line 99, in top-level template code
jinja2.exceptions.UndefinedError: the template 'filebeat/map.jinja' (imported on line 25) does not export the requested name 'SO'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable the template 'filebeat/map.jinja' (imported on line 25) does not export the requested name 'SO'
[CRITICAL] Rendering SLS 'base:filebeat' failed: Jinja variable the template 'filebeat/map.jinja' (imported on line 25) does not export the requested name 'SO'
local:
Data failed to compile:

Rendering SLS 'base:filebeat' failed: Jinja variable the template 'filebeat/map.jinja' (imported on line 25) does not export the requested name 'SO'

---

I tried "sudo so-docker-refresh" which reports for filebeat :

Downloading so-filebeat
Executing command with retry support: docker pull ghcr.io/security-onion-solutio ns/so-filebeat:2.3.140
Results: 2.3.140: Pulling from security-onion-solutions/so-filebeat
Digest: sha256:8fa3fcf470db437c81b60a4c933a71c7268459724e3de7036e1041f12968dc43
Status: Image is up to date for ghcr.io/security-onion-solutions/so-filebeat:2.3 .140
ghcr.io/security-onion-solutions/so-filebeat:2.3.140 (0)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 543 100 543 0 0 1450 0 --:--:-- --:--:-- --:--:-- 1448
The push refers to repository [so:5000/security-onion-solutions/so-filebeat]
411b5bb337f8: Layer already exists
6035df72c737: Layer already exists
e73c3a9e97dd: Layer already exists
1dff359cd69c: Layer already exists
443e8dbe8511: Layer already exists
d41a615b231f: Layer already exists
3a72beac0431: Layer already exists
f1c4a66ee9c4: Layer already exists
f8d067aea552: Layer already exists
8593ba51c151: Layer already exists
afbf51b5b036: Layer already exists
b11d6b9b71d1: Layer already exists
72fa1596fd90: Layer already exists
347aad7cccc0: Layer already exists
7d0baa64e0f6: Layer already exists
af7ed92504ae: Layer already exists
2.3.140: digest: sha256:7e0240795eeabeace2a924084b435c36ea100a846bf59d770980c2f1 fd249549 size: 3658

---

[btdewey]

This isn't helpful, but the error "No such container: so-filebeat" probably just means that the container hasn't started so it doesn't exist. You get the same error when trying to restart a container that isn't running (it will error trying to stop the container because it doesn't exist then it will start the container).
This is likely the error you really want to investigate: "jinja2.exceptions.UndefinedError: the template 'filebeat/map.jinja' (imported on line 25) does not export the requested name 'SO'"
I suspect this jinja file has an error in it, but other than getting a working one from a different system and comparing I don't have any suggestions about what the issue could be.

[rabeyer]

Thanks for the response btdewey. I did check the "filebeat/map.jinja" to the version repo and it matches so it's a deeper issue. I'm getting close to doing a total reload since I'm not seeing any good ideas here. This is a carryover from discussion #8340 that received no good solution. It was closed without a real answer. The only mods made were to the so_standalone.sls and not the init.sls by either the initiator of that discussion or me. I removed my small changes to activate the Fortigate parsing as suggested and still have the same issue. Bummer.

[weslambert]

What is the output of the following?

ls -la /opt/so/saltstack/local/salt/filebeat/*

[rabeyer]

The only changes to global.sls was under steno: and nids: to ignore traffic to our cloud backup IPs. The change to the minion so_standalone.sls for parsing Fortigate logs was removed. I still do have a list of sids to disable under idstools

[weslambert]

What is the output of the following?

md5sum /opt/so/saltstack/default/salt/filebeat/*

[rabeyer]

md5sum: /opt/so/saltstack/default/salt/filebeat/etc: Is a directory
38b9f26579d4bf2d327841b96cc1a7bc /opt/so/saltstack/default/salt/filebeat/init.sls
c1a29a67ef82586003236df3372842b7 /opt/so/saltstack/default/salt/filebeat/map.jinja
e5b32b1258191f0e957008f689fc3b68 /opt/so/saltstack/default/salt/filebeat/modules.map.jinja
8a8f72ee9a252aca8088b4360da12b4c /opt/so/saltstack/default/salt/filebeat/pillar.example
8a19b7ddc7e0ecc866d3b6e917d93225 /opt/so/saltstack/default/salt/filebeat/securityoniondefaults.yaml
555e6e5e60d99378665d12f8a4c41373 /opt/so/saltstack/default/salt/filebeat/thirdpartydefaults.yaml

[weslambert]

It seems like your hashes are the same as the latest code. I'm not sure how it would still be trying to hold onto old config unless there is another copy being referenced somewhere, or it is being cached. You can compare against the files here, if needed:

https://github.com/Security-Onion-Solutions/securityonion/tree/master/salt/filebeat

[rabeyer]

I guess we're at an impasse and the only option at this point is to do a clean reload. That will be a project for next week. I guess this taught me a big lesson that I'll need to: 1) figure out a system to do a full backup before I do an update and 2) wait a few weeks after a new version to do the update.

Thanks for all the time you put into this. Some things in life are a mystery...I guess this is one of those.

[rabeyer]

@weslambert I found the solution. The cache suggestion you gave was key. I did a global find for */filebeat/init.sls and found 5 instances.

There is one at /var/cache/salt/minion/files/base/filebeat/init.sls that did have the line
{% from 'filebeat/map.jinja' import SO with context %}

I deleted it, ran the so-filebeat-start and it errored, as usual. The file that came back to the cache still had the bad line. So, the key to fixing this was finding that there is an init.sls in /opt/so/saltstack/local/salt/filebeat/ that had the bad line. Apparently, it is not updated by soup. Entering the commands below fixed it:

sudo -i
cp /opt/so/saltstack/default/salt/filebeat/init.sls /opt/so/saltstack/local/salt/filebeat/init.sls
rm /var/cache/salt/minion/files/base/filebeat/init.sls

After that, filebeat started and I'm all green on the containers after a reboot.

Again, thanks for sticking with me on this.

[rabeyer]

No harm...it got me to put on my thinking cap and get a little more familiar with how things work. I really do appreciate all of the effort you put into helping me.

[weslambert]

No problem! 👍

[EbolaWare]

In the future, you can dump cache on your minions using a salt '*' saltutil.clear_cache command. Found here .

[rabeyer]

Thanks! Good to know.

[BangAbe88]

Thanks u brother, youre the BEST