so-soc MISSING

[kiaraho]

I was testing the new Security Onion 2 in a distributed environment with one Manager, Search and Forward node each. The first time I setup the environment I noticed so-soc went missing after a few hours. Believing I did something wrong, I rebuilt the entire environment from scratch by reinstalling Ubuntu 18.04 then running so-setup-network. Once again, everything worked fine for a few hours, then so-soc went missing once more.

soadmin@so2-manager:~$ sudo so-status

Checking Docker status

    Docker ------------------------------------------------------------------------------------------ [ OK ]    

Checking container statuses

    so-aptcacherng ----------------------------------------------------------------------------- [ MISSING ]    
    so-cortex --------------------------------------------------------------------------------------- [ OK ]    
    so-dockerregistry ------------------------------------------------------------------------------- [ OK ]    
    so-elastalert ----------------------------------------------------------------------------------- [ OK ]    
    so-elasticsearch -------------------------------------------------------------------------------- [ OK ]    
    so-filebeat ------------------------------------------------------------------------------------- [ OK ]    
    so-fleet ---------------------------------------------------------------------------------------- [ OK ]    
    so-grafana -------------------------------------------------------------------------------------- [ OK ]    
    so-idstools ------------------------------------------------------------------------------------- [ OK ]    
    so-influxdb ------------------------------------------------------------------------------------- [ OK ]    
    so-kibana --------------------------------------------------------------------------------------- [ OK ]    
    so-kratos --------------------------------------------------------------------------------------- [ OK ]    
    so-logstash ------------------------------------------------------------------------------------- [ OK ]    
    so-mysql ---------------------------------------------------------------------------------------- [ OK ]    
    so-nginx ---------------------------------------------------------------------------------------- [ OK ]    
    so-playbook ------------------------------------------------------------------------------------- [ OK ]    
    so-redis ---------------------------------------------------------------------------------------- [ OK ]    
    so-soc ------------------------------------------------------------------------------------- [ MISSING ]    
    so-soctopus ------------------------------------------------------------------------------------- [ OK ]    
    so-telegraf ------------------------------------------------------------------------------------- [ OK ]    
    so-thehive -------------------------------------------------------------------------------------- [ OK ]    
    so-thehive-es ----------------------------------------------------------------------------------- [ OK ]    
    so-wazuh ---------------------------------------------------------------------------------------- [ OK ]

soadmin@so2-manager:~$ curl -k -s https://localhost:5000/v2/securityonion\/so-soc/tags/list | jq -r .
{
  "name": "securityonion/so-soc",
  "tags": [
    "2.3.1"
  ]
}

Running so-docker-refresh and rebooting the Manager had no effect on the missing modules.

[TOoSmOotH]

What is in the log in /opt/so/log/soc?

[kiaraho]

There is a /opt/so/log/soc/sensoroni-server.log file that has 931 lines. Any specific errors I should be looking for?

[TOoSmOotH]

I believe you can attach it

[kiaraho]

sensoroni-server.log

[kiaraho]

Additionally, here is the output of the so-soc-restart command.

soadmin@so2-manager:~$ sudo so-soc-restart
=========================================================================
Restarting soc...

This could take a while if another Salt job is running. 
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
Error response from daemon: No such container: so-soc
Error: No such container: so-soc

[TOoSmOotH]

sudo docker images

[kiaraho]

soadmin@so2-manager:~$ sudo docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
securityonion/so-soc                                   2.3.1               a37e4d7ff9ed        4 days ago          59.8MB
so2-manager:5000/securityonion/so-soc                  2.3.1               a37e4d7ff9ed        4 days ago          59.8MB
securityonion/so-strelka-manager                       2.3.1               6e4bb2ca8a38        5 days ago          20.8MB
so2-manager:5000/securityonion/so-strelka-manager      2.3.1               6e4bb2ca8a38        5 days ago          20.8MB
securityonion/so-strelka-frontend                      2.3.1               7e27eee1f26d        5 days ago          21.3MB
so2-manager:5000/securityonion/so-strelka-frontend     2.3.1               7e27eee1f26d        5 days ago          21.3MB
securityonion/so-curator                               2.3.1               ddf2d268e32c        6 days ago          123MB
so2-manager:5000/securityonion/so-curator              2.3.1               ddf2d268e32c        6 days ago          123MB
securityonion/so-grafana                               2.3.1               3afe8c9fb345        6 days ago          475MB
so2-manager:5000/securityonion/so-grafana              2.3.1               3afe8c9fb345        6 days ago          475MB
securityonion/so-steno                                 2.3.1               9d68f2f2eff8        6 days ago          998MB
so2-manager:5000/securityonion/so-steno                2.3.1               9d68f2f2eff8        6 days ago          998MB
securityonion/so-redis                                 2.3.1               b356bb4e8998        6 days ago          32.2MB
so2-manager:5000/securityonion/so-redis                2.3.1               b356bb4e8998        6 days ago          32.2MB
securityonion/so-soctopus                              2.3.1               a495568b960b        6 days ago          598MB
so2-manager:5000/securityonion/so-soctopus             2.3.1               a495568b960b        6 days ago          598MB
securityonion/so-mysql                                 2.3.1               6a66f08e3fb8        6 days ago          250MB
so2-manager:5000/securityonion/so-mysql                2.3.1               6a66f08e3fb8        6 days ago          250MB
so2-manager:5000/securityonion/so-nginx                2.3.1               34aa8503eeb2        6 days ago          84.8MB
securityonion/so-nginx                                 2.3.1               34aa8503eeb2        6 days ago          84.8MB
securityonion/so-zeek                                  2.3.1               0ea6077f3f96        6 days ago          1.13GB
so2-manager:5000/securityonion/so-zeek                 2.3.1               0ea6077f3f96        6 days ago          1.13GB
securityonion/so-kibana                                2.3.1               5bc490d06bac        6 days ago          1.62GB
so2-manager:5000/securityonion/so-kibana               2.3.1               5bc490d06bac        6 days ago          1.62GB
securityonion/so-fleet-launcher                        2.3.1               152debdfb55a        6 days ago          558MB
so2-manager:5000/securityonion/so-fleet-launcher       2.3.1               152debdfb55a        6 days ago          558MB
securityonion/so-filebeat                              2.3.1               7ef2a908e053        6 days ago          443MB
so2-manager:5000/securityonion/so-filebeat             2.3.1               7ef2a908e053        6 days ago          443MB
securityonion/so-pcaptools                             2.3.1               6b6f87408cd4        6 days ago          120MB
so2-manager:5000/securityonion/so-pcaptools            2.3.1               6b6f87408cd4        6 days ago          120MB
securityonion/so-thehive                               2.3.1               25e149d12260        6 days ago          603MB
so2-manager:5000/securityonion/so-thehive              2.3.1               25e149d12260        6 days ago          603MB
so2-manager:5000/securityonion/so-suricata             2.3.1               e62b2125198d        6 days ago          694MB
securityonion/so-suricata                              2.3.1               e62b2125198d        6 days ago          694MB
securityonion/so-thehive-cortex                        2.3.1               fb03715ad7a7        6 days ago          1.56GB
so2-manager:5000/securityonion/so-thehive-cortex       2.3.1               fb03715ad7a7        6 days ago          1.56GB
securityonion/so-fleet                                 2.3.1               5f91ea8d9eed        6 days ago          70.7MB
so2-manager:5000/securityonion/so-fleet                2.3.1               5f91ea8d9eed        6 days ago          70.7MB
securityonion/so-logstash                              2.3.1               67e9b5e451f2        6 days ago          1.34GB
so2-manager:5000/securityonion/so-logstash             2.3.1               67e9b5e451f2        6 days ago          1.34GB
securityonion/so-strelka-filestream                    2.3.1               fdc5cfd98a58        6 days ago          20MB
so2-manager:5000/securityonion/so-strelka-filestream   2.3.1               fdc5cfd98a58        6 days ago          20MB
so2-manager:5000/securityonion/so-acng                 2.3.1               866ecc0b687c        6 days ago          70.2MB
securityonion/so-acng                                  2.3.1               866ecc0b687c        6 days ago          70.2MB
securityonion/so-minio                                 2.3.1               780978a9e32e        6 days ago          59MB
so2-manager:5000/securityonion/so-minio                2.3.1               780978a9e32e        6 days ago          59MB
securityonion/so-idstools                              2.3.1               d48b723f4169        6 days ago          55.7MB
so2-manager:5000/securityonion/so-idstools             2.3.1               d48b723f4169        6 days ago          55.7MB
securityonion/so-kratos                                2.3.1               6b4d628666cc        6 days ago          50.3MB
so2-manager:5000/securityonion/so-kratos               2.3.1               6b4d628666cc        6 days ago          50.3MB
securityonion/so-strelka-backend                       2.3.1               a5369c1bff98        6 days ago          753MB
so2-manager:5000/securityonion/so-strelka-backend      2.3.1               a5369c1bff98        6 days ago          753MB
securityonion/so-domainstats                           2.3.1               0f02de72fd5b        6 days ago          110MB
so2-manager:5000/securityonion/so-domainstats          2.3.1               0f02de72fd5b        6 days ago          110MB
securityonion/so-freqserver                            2.3.1               78e84771cc3f        6 days ago          85.7MB
so2-manager:5000/securityonion/so-freqserver           2.3.1               78e84771cc3f        6 days ago          85.7MB
securityonion/so-wazuh                                 2.3.1               99f60045d7ef        6 days ago          881MB
so2-manager:5000/securityonion/so-wazuh                2.3.1               99f60045d7ef        6 days ago          881MB
securityonion/so-elastalert                            2.3.1               1a5b57a35105        6 days ago          145MB
so2-manager:5000/securityonion/so-elastalert           2.3.1               1a5b57a35105        6 days ago          145MB
securityonion/so-influxdb                              2.3.1               c2263941273a        6 days ago          117MB
so2-manager:5000/securityonion/so-influxdb             2.3.1               c2263941273a        6 days ago          117MB
securityonion/so-elasticsearch                         2.3.1               d7b7081c82f7        6 days ago          1.07GB
so2-manager:5000/securityonion/so-elasticsearch        2.3.1               d7b7081c82f7        6 days ago          1.07GB
securityonion/so-thehive-es                            2.3.1               e68a77b5ad1a        6 days ago          748MB
so2-manager:5000/securityonion/so-thehive-es           2.3.1               e68a77b5ad1a        6 days ago          748MB
so2-manager:5000/securityonion/so-telegraf             2.3.1               632dfa6f9e8b        6 days ago          80.5MB
securityonion/so-telegraf                              2.3.1               632dfa6f9e8b        6 days ago          80.5MB
securityonion/so-playbook                              2.3.1               c0ad97006462        6 days ago          629MB
so2-manager:5000/securityonion/so-playbook             2.3.1               c0ad97006462        6 days ago          629MB
registry                                               latest              2d4f4b5309b1        4 months ago        26.2MB

[TOoSmOotH]

Update to 2.3.2 and see if this clears it up. I don't see anything in the log that seems to point to an issue.

[kiaraho]

Ran sudo soup, waited for Salt to push updates to the Search and Forward nodes, then rebooted the Manager. Still no luck.

soadmin@so2-manager:~$ sudo docker images
REPOSITORY                                         TAG                 IMAGE ID            CREATED             SIZE
so2-manager:5000/securityonion/so-kibana           2.3.2               c982964e4e1b        29 hours ago        1.62GB
so2-manager:5000/securityonion/so-logstash         2.3.2               66ba23003e12        30 hours ago        1.34GB
so2-manager:5000/securityonion/so-elasticsearch    2.3.2               45b63c4e0be9        30 hours ago        1.03GB
so2-manager:5000/securityonion/so-filebeat         2.3.2               214c48474db0        30 hours ago        443MB
so2-manager:5000/securityonion/so-grafana          2.3.2               3afe8c9fb345        6 days ago          475MB
so2-manager:5000/securityonion/so-redis            2.3.2               b356bb4e8998        6 days ago          32.2MB
so2-manager:5000/securityonion/so-soctopus         2.3.2               a495568b960b        6 days ago          598MB
so2-manager:5000/securityonion/so-mysql            2.3.2               6a66f08e3fb8        6 days ago          250MB
so2-manager:5000/securityonion/so-nginx            2.3.2               34aa8503eeb2        6 days ago          84.8MB
so2-manager:5000/securityonion/so-kibana           2.3.1               5bc490d06bac        6 days ago          1.62GB
so2-manager:5000/securityonion/so-filebeat         2.3.1               7ef2a908e053        6 days ago          443MB
so2-manager:5000/securityonion/so-thehive          2.3.2               25e149d12260        6 days ago          603MB
so2-manager:5000/securityonion/so-thehive-cortex   2.3.2               fb03715ad7a7        6 days ago          1.56GB
so2-manager:5000/securityonion/so-fleet            2.3.2               5f91ea8d9eed        6 days ago          70.7MB
so2-manager:5000/securityonion/so-logstash         2.3.1               67e9b5e451f2        6 days ago          1.34GB
so2-manager:5000/securityonion/so-idstools         2.3.2               d48b723f4169        6 days ago          55.7MB
so2-manager:5000/securityonion/so-kratos           2.3.2               6b4d628666cc        6 days ago          50.3MB
so2-manager:5000/securityonion/so-wazuh            2.3.2               99f60045d7ef        6 days ago          881MB
so2-manager:5000/securityonion/so-elastalert       2.3.2               1a5b57a35105        6 days ago          145MB
so2-manager:5000/securityonion/so-influxdb         2.3.2               c2263941273a        6 days ago          117MB
so2-manager:5000/securityonion/so-elasticsearch    2.3.1               d7b7081c82f7        6 days ago          1.07GB
so2-manager:5000/securityonion/so-thehive-es       2.3.2               e68a77b5ad1a        6 days ago          748MB
so2-manager:5000/securityonion/so-telegraf         2.3.2               632dfa6f9e8b        6 days ago          80.5MB
so2-manager:5000/securityonion/so-playbook         2.3.2               c0ad97006462        6 days ago          629MB
registry                                           latest              2d4f4b5309b1        4 months ago        26.2MB

soadmin@so2-manager:~$ sudo so-status

Checking Docker status

    Docker ------------------------------------------------------------------------------------------------- [ OK ]    

Checking container statuses

    so-aptcacherng ------------------------------------------------------------------------------------ [ MISSING ]    
    so-cortex ------------------------------------------------------------------------------------------- [ ERROR ]    
    so-dockerregistry -------------------------------------------------------------------------------------- [ OK ]    
    so-elastalert --------------------------------------------------------------------------------------- [ ERROR ]    
    so-elasticsearch --------------------------------------------------------------------------------------- [ OK ]    
    so-filebeat -------------------------------------------------------------------------------------------- [ OK ]    
    so-fleet -------------------------------------------------------------------------------------------- [ ERROR ]    
    so-grafana ------------------------------------------------------------------------------------------ [ ERROR ]    
    so-idstools ----------------------------------------------------------------------------------------- [ ERROR ]    
    so-influxdb ----------------------------------------------------------------------------------------- [ ERROR ]    
    so-kibana ---------------------------------------------------------------------------------------------- [ OK ]    
    so-kratos ------------------------------------------------------------------------------------------- [ ERROR ]    
    so-logstash -------------------------------------------------------------------------------------------- [ OK ]    
    so-mysql -------------------------------------------------------------------------------------------- [ ERROR ]    
    so-nginx -------------------------------------------------------------------------------------------- [ ERROR ]    
    so-playbook ----------------------------------------------------------------------------------------- [ ERROR ]    
    so-redis -------------------------------------------------------------------------------------------- [ ERROR ]    
    so-soc -------------------------------------------------------------------------------------------- [ MISSING ]    
    so-soctopus ----------------------------------------------------------------------------------------- [ ERROR ]    
    so-telegraf ----------------------------------------------------------------------------------------- [ ERROR ]    
    so-thehive ------------------------------------------------------------------------------------------ [ ERROR ]    
    so-thehive-es --------------------------------------------------------------------------------------- [ ERROR ]    
    so-wazuh -------------------------------------------------------------------------------------------- [ ERROR ]   

Since this is a test environment, I'll wipe it and start again. Maybe the third time is the charm. Though I'm still concerned about what caused the issue in the first place

[kiaraho]

So I rebuilt the entire environment (Manager, SearchNode, ForwardNode) from scratch. Everything worked fine for about 24hrs. I then rebooted the Manager after which so-soc reported an ERROR state. I then ran so-soc-restart and the status switched from ERROR to MISSING once again. No other changes were made to the environment.

soadmin@so2-manager:~$ sudo so-soc-restart
=========================================================================
Restarting soc...

This could take a while if another Salt job is running. 
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
so-soc
so-soc
[ERROR   ] Exception during resolving address: [Errno 2] Host name lookup failure
[ERROR   ] Exception during resolving address: [Errno 2] Host name lookup failure
[ERROR   ] Exception during resolving address: [Errno 2] Host name lookup failure
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 400, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 36, in top-level template code
jinja2.exceptions.UndefinedError: the inline if-expression on line 36 evaluated to false and no else section was defined.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 166, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 407, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {0}{1}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable the inline if-expression on line 36 evaluated to false and no else section was defined.
[ERROR   ] Unable to manage file: Jinja variable the inline if-expression on line 36 evaluated to false and no else section was defined.

      Result: False
     Comment: One or more requisite failed: soc.socconfig
     Started: 15:34:56.034655
    Duration: 0.019 ms
     Changes:   
----------
          ID: kratosgroup
    Function: group.present
        Name: kratos
      Result: True
     Comment: Group kratos is present and up to date
     Started: 15:34:56.035848
    Duration: 7.803 ms
     Changes:   
----------
          ID: kratos
    Function: user.present
      Result: True
     Comment: User kratos is present and up to date
     Started: 15:34:56.044599
    Duration: 25.34 ms
     Changes:   
----------
          ID: kratosdir
    Function: file.directory
        Name: /opt/so/conf/kratos/db
      Result: True
     Comment: The directory /opt/so/conf/kratos/db is in the correct state
     Started: 15:34:56.070216
    Duration: 3.026 ms
     Changes:   
----------
          ID: kratoslogdir
    Function: file.directory
        Name: /opt/so/log/kratos
      Result: True
     Comment: The directory /opt/so/log/kratos is in the correct state
     Started: 15:34:56.073479
    Duration: 2.81 ms
     Changes:   
----------
          ID: kratossync
    Function: file.recurse
        Name: /opt/so/conf/kratos
      Result: True
     Comment: The directory /opt/so/conf/kratos is in the correct state
     Started: 15:34:56.076466
    Duration: 90.05 ms
     Changes:   
----------
          ID: so-kratos
    Function: docker_container.running
      Result: True
     Comment: Container 'so-kratos' is already configured as specified
     Started: 15:34:56.167352
    Duration: 2049.737 ms
     Changes:   

Summary for local
-------------
Succeeded: 10
Failed:     2
-------------
Total states run:     12
Total run time:    2.351 s

soadmin@so2-manager:~$ sudo salt '*' so.status
so2-searchnode_searchnode:
    
    Checking Docker status
    
        Docker ----------------------- [ OK ]    
    
    Checking container statuses
    
        so-curator ------------------- [ OK ]    
        so-elasticsearch ------------- [ OK ]    
        so-filebeat ------------------ [ OK ]    
        so-logstash ------------------ [ OK ]    
        so-nginx --------------------- [ OK ]    
        so-telegraf ------------------ [ OK ]    
        so-wazuh --------------------- [ OK ]
so2-forwardnode_sensor:
    
    Checking Docker status
    
        Docker ----------------------- [ OK ]    
    
    Checking container statuses
    
        so-filebeat ------------------ [ OK ]    
        so-sensoroni ----------------- [ OK ]    
        so-steno --------------------- [ OK ]    
        so-strelka-coordinator ------- [ OK ]    
        so-strelka-filestream -------- [ OK ]    
        so-strelka-frontend ---------- [ OK ]    
        so-strelka-gatekeeper -------- [ OK ]    
        so-strelka-manager ----------- [ OK ]    
        so-suricata ------------------ [ OK ]    
        so-telegraf ------------------ [ OK ]    
        so-zeek ---------------------- [ OK ]
so2-manager_manager:
    
    Checking Docker status
    
        Docker ----------------------- [ OK ]    
    
    Checking container statuses
    
        so-aptcacherng ---------- [ MISSING ]    
        so-cortex -------------------- [ OK ]    
        so-dockerregistry ------------ [ OK ]    
        so-elastalert ---------------- [ OK ]    
        so-elasticsearch ------------- [ OK ]    
        so-filebeat ------------------ [ OK ]    
        so-fleet --------------------- [ OK ]    
        so-grafana ------------------- [ OK ]    
        so-idstools ------------------ [ OK ]    
        so-influxdb ------------------ [ OK ]    
        so-kibana -------------------- [ OK ]    
        so-kratos -------------------- [ OK ]    
        so-logstash ------------------ [ OK ]    
        so-mysql --------------------- [ OK ]    
        so-nginx --------------------- [ OK ]    
        so-playbook ------------------ [ OK ]    
        so-redis --------------------- [ OK ]    
        so-soc ------------------ [ MISSING ]    
        so-soctopus ------------------ [ OK ]    
        so-telegraf ------------------ [ OK ]    
        so-thehive ------------------- [ OK ]    
        so-thehive-es ---------------- [ OK ]    
        so-wazuh --------------------- [ OK ]

[jertel]

The fix for the above issue was merged in this week and will go out with our next planned release. You can see the one-line fix here: #1568

[kiaraho]

@jertel, I only have one SearchNode in my test environment. Does issue #1568 also apply in this scenario?

[jertel]

Yes, the description is not entirely accurate. If at least one search node exists you will encounter this issue.

[kiaraho]

Will the fix require an entire rebuild of the Manager, or just updating to 2.3.3?

[jertel]

It will be corrected during the update to the next planned release. If you'd like to manually patch 2.3.2 in the meantime you can make the change shown in the PR to this file on your manager: /opt/so/saltstack/default/salt/soc/files/soc/soc.json and then salt-call state.apply soc queue=True