so-soc container missing

[Gregg3]

Version

2.4.90

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

28

RAM

395567560

Storage for /

293G

Storage for /nsm

1.2T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

We have a Manager node and a Forward node.
The so-soc container shows as 'missing' on the manager node even though we've run so-docker-refresh. I'm pretty sure that I caused the problem. When we installed Security Onion 2.4, we set it up to use an internet proxy. Last month, I attempted to change SO to direct (NAT) internet communications - but I must have goofed somewhere.

                          Security Onion Status
                         Container │ Status  │                    Details
───────────────────────────────────┼─────────┼────────────────────────────
                 so-dockerregistry │ running │           Up About an hour
                     so-elastalert │ running │           Up About an hour
                  so-elastic-fleet │ running │           Up About an hour
 so-elastic-fleet-package-registry │ running │ Up About an hour (healthy)
                  so-elasticsearch │ running │           Up About an hour
                       so-idstools │ running │           Up About an hour
                       so-influxdb │ running │ Up About an hour (healthy)
                         so-kibana │ running │           Up About an hour
                         so-kratos │ running │           Up About an hour
                       so-logstash │ running │           Up About an hour
                          so-nginx │ running │ Up About an hour (healthy)
                          so-redis │ running │           Up About an hour
                      so-sensoroni │ running │           Up About an hour
                            so-soc │ missing │
                       so-telegraf │ running │           Up About an hour

 ❗ Check container logs and /opt/so/log for more details.

"sudo salt-call state.highstate" appears to check out okay. it ends with:

Summary for local
--------------
Succeeded: 733 (changed=33)
Failed:      0
--------------
Total states run:     733
Total run time:    95.740 s

When I start the so-soc container, it seems to start without any errors:

Summary for local
-------------
Succeeded: 41 (changed=2)
Failed:     0
-------------
Total states run:     41
Total run time:   12.971 s

but the docker logs show this:

[root@so-manager log]# docker logs so-soc
2024/08/27 19:33:19 error Unmarshal error reading json object error=json: cannot unmarshal number into Go struct field ServerConfig.server.proxy of type string offset=268965
Error: Unable to read configuration file 'sensoroni.json' [json: cannot unmarshal number into Go struct field ServerConfig.server.proxy of type string]
[root@so-manager log]#

Where is it grabbing 'ServerConfig.server.proxy' from?
What json file do I need to fix?

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[Gregg3]

I haven't found the solution yet, but I did upgrade.
/etc/soversion is now 2.4.100
so-soc is still 'missing'

[nebra-status]

Did you verify sensorino.log in opt/so/log/soc to see if there any errors?

[Gregg3]

Thank you for taking the time to respond to my problem.
Unfortunately the /opt/so/log/soc/sensoroni-server.log file is empty, as well as the 14 previous versions of that log file that are retained by logrotate.

Are there any options for verbose logging when starting up so-soc that might help resolve my problem?

[Gregg3]

Is the 404 response from Kratos an indication of a problem even though the status of so-kratos is 'running'

[root@so-manager soc]# so-soc-start
=========================================================================
Starting soc...

This could take a while if another Salt job is running.
Run this command with --force to stop all Salt jobs before proceeding.
=========================================================================
[INFO    ] Loading fresh modules for state activity
[INFO    ] Running state [/opt/so/saltstack/local/pillar/elasticsearch/auth.sls] at time 12:58:08.187941
[INFO    ] Executing state file.managed for [/opt/so/saltstack/local/pillar/elasticsearch/auth.sls]
[INFO    ] File /opt/so/saltstack/local/pillar/elasticsearch/auth.sls is in the correct state
[INFO    ] Completed state [/opt/so/saltstack/local/pillar/elasticsearch/auth.sls] at time 12:58:13.172578 (duration_in
ms=4984.637)
[INFO    ] Running state [kratos] at time 12:58:13.173951
[INFO    ] Executing state group.present for [kratos]
[INFO    ] Group kratos is present and up to date
[INFO    ] Completed state [kratos] at time 12:58:13.179250 (duration_in_ms=5.298)
[INFO    ] Running state [kratos] at time 12:58:13.179982
[INFO    ] Executing state user.present for [kratos]
[INFO    ] User kratos is present and up to date
[INFO    ] Completed state [kratos] at time 12:58:13.192643 (duration_in_ms=12.661)
[INFO    ] Running state [/nsm/kratos] at time 12:58:13.192921
[INFO    ] Executing state file.directory for [/nsm/kratos]
[INFO    ] The directory /nsm/kratos is in the correct state
[INFO    ] Completed state [/nsm/kratos] at time 12:58:13.194599 (duration_in_ms=1.678)
[INFO    ] Running state [/nsm/kratos/db] at time 12:58:13.194810
[INFO    ] Executing state file.directory for [/nsm/kratos/db]
[INFO    ] The directory /nsm/kratos/db is in the correct state
[INFO    ] Completed state [/nsm/kratos/db] at time 12:58:13.196081 (duration_in_ms=1.272)
[INFO    ] Running state [/opt/so/log/kratos] at time 12:58:13.196288
[INFO    ] Executing state file.directory for [/opt/so/log/kratos]
[INFO    ] The directory /opt/so/log/kratos is in the correct state
[INFO    ] Completed state [/opt/so/log/kratos] at time 12:58:13.197588 (duration_in_ms=1.299)
[INFO    ] Running state [/opt/so/conf/kratos/schema.json] at time 12:58:13.197873
[INFO    ] Executing state file.managed for [/opt/so/conf/kratos/schema.json]
[INFO    ] File /opt/so/conf/kratos/schema.json is in the correct state
[INFO    ] Completed state [/opt/so/conf/kratos/schema.json] at time 12:58:13.210537 (duration_in_ms=12.663)
[INFO    ] Running state [/opt/so/conf/kratos/oidc.jsonnet] at time 12:58:13.210800
[INFO    ] Executing state file.managed for [/opt/so/conf/kratos/oidc.jsonnet]
[INFO    ] File /opt/so/conf/kratos/oidc.jsonnet is in the correct state
[INFO    ] Completed state [/opt/so/conf/kratos/oidc.jsonnet] at time 12:58:13.222695 (duration_in_ms=11.895)
[INFO    ] Running state [/opt/so/conf/kratos/kratos.yaml] at time 12:58:13.222966
[INFO    ] Executing state file.managed for [/opt/so/conf/kratos/kratos.yaml]
[INFO    ] File /opt/so/conf/kratos/kratos.yaml is in the correct state
[INFO    ] Completed state [/opt/so/conf/kratos/kratos.yaml] at time 12:58:13.245429 (duration_in_ms=22.463)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:58:13.245677
[INFO    ] Executing state file.append for [/opt/so/conf/so-status/so-status.conf]
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop import
ng 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability laye
. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
[INFO    ] Executing command git in directory '/root'
[INFO    ] Executing command 'grep' in directory '/root'
[INFO    ] ['unless condition is true']
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:58:14.686830 (duration_in_ms=1441.152)
[INFO    ] Running state [so-kratos] at time 12:58:14.688360
[INFO    ] Executing state docker_container.running for [so-kratos]
[INFO    ] Container 'so-kratos' is already configured as specified
[INFO    ] Completed state [so-kratos] at time 12:58:16.055893 (duration_in_ms=1367.533)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:58:16.056163
[INFO    ] Executing state file.uncomment for [/opt/so/conf/so-status/so-status.conf]
[INFO    ] Pattern already uncommented
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:58:16.058666 (duration_in_ms=2.502)
[INFO    ] Running state [http://so-manager:4434/] at time 12:58:16.059187
[INFO    ] Executing state http.wait_for_successful_query for [http://so-manager:4434/]
[INFO    ] Status 404 was found.
[INFO    ] Completed state [http://so-manager:4434/] at time 12:58:16.066425 (duration_in_ms=7.238)
[INFO    ] Running state [/var/tmp/so-user.lock] at time 12:58:16.066643
[INFO    ] Executing state file.missing for [/var/tmp/so-user.lock]
[INFO    ] Path /var/tmp/so-user.lock is missing
[INFO    ] Completed state [/var/tmp/so-user.lock] at time 12:58:16.067470 (duration_in_ms=0.827)
[INFO    ] Running state [so-user sync] at time 12:58:16.071699
[INFO    ] Executing state cmd.run for [so-user sync]
[INFO    ] ['All files in creates exist']
[INFO    ] Completed state [so-user sync] at time 12:58:16.079448 (duration_in_ms=7.748)
[INFO    ] Running state [PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-user
sync &>> /opt/so/log/soc/sync.log] at time 12:58:16.079945
[INFO    ] Executing state cron.present for [PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bi
 /usr/sbin/so-user sync &>> /opt/so/log/soc/sync.log]
[INFO    ] Executing command 'grep' in directory '/root'
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Cron PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-user sync &>>
opt/so/log/soc/sync.log already present
[INFO    ] Completed state [PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-us
r sync &>> /opt/so/log/soc/sync.log] at time 12:58:16.095029 (duration_in_ms=15.084)
[INFO    ] Running state [/opt/so/conf/sigma/repos] at time 12:58:16.095272
[INFO    ] Executing state file.directory for [/opt/so/conf/sigma/repos]
[INFO    ] The directory /opt/so/conf/sigma/repos is in the correct state
[INFO    ] Completed state [/opt/so/conf/sigma/repos] at time 12:58:16.096780 (duration_in_ms=1.508)
[INFO    ] Running state [/opt/so/rules/elastalert/rules] at time 12:58:16.097009
[INFO    ] Executing state file.directory for [/opt/so/rules/elastalert/rules]
[INFO    ] The directory /opt/so/rules/elastalert/rules is in the correct state
[INFO    ] Completed state [/opt/so/rules/elastalert/rules] at time 12:58:16.098332 (duration_in_ms=1.323)
[INFO    ] Running state [/opt/so/conf/soc/fingerprints] at time 12:58:16.098546
[INFO    ] Executing state file.directory for [/opt/so/conf/soc/fingerprints]
[INFO    ] The directory /opt/so/conf/soc/fingerprints is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/fingerprints] at time 12:58:16.099808 (duration_in_ms=1.262)
[INFO    ] Running state [/nsm/soc/jobs] at time 12:58:16.100025
[INFO    ] Executing state file.directory for [/nsm/soc/jobs]
[INFO    ] The directory /nsm/soc/jobs is in the correct state
[INFO    ] Completed state [/nsm/soc/jobs] at time 12:58:16.101372 (duration_in_ms=1.346)
[INFO    ] Running state [/opt/so/log/soc] at time 12:58:16.101599
[INFO    ] Executing state file.directory for [/opt/so/log/soc]
[INFO    ] The directory /opt/so/log/soc is in the correct state
[INFO    ] Completed state [/opt/so/log/soc] at time 12:58:16.107451 (duration_in_ms=5.851)
[INFO    ] Running state [/opt/so/conf/soc/queue] at time 12:58:16.107673
[INFO    ] Executing state file.directory for [/opt/so/conf/soc/queue]
[INFO    ] The directory /opt/so/conf/soc/queue is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/queue] at time 12:58:16.108937 (duration_in_ms=1.264)
[INFO    ] Running state [/opt/so/conf/soc/analytics.js] at time 12:58:16.109164
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/analytics.js]
[INFO    ] File /opt/so/conf/soc/analytics.js is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/analytics.js] at time 12:58:16.122684 (duration_in_ms=13.519)
[INFO    ] Running state [/opt/so/conf/soc/soc.json] at time 12:58:16.122918
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/soc.json]
[INFO    ] File /opt/so/conf/soc/soc.json is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/soc.json] at time 12:58:18.466187 (duration_in_ms=2343.268)
[INFO    ] Running state [/opt/so/conf/soc/motd.md] at time 12:58:18.466583
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/motd.md]
[INFO    ] File /opt/so/conf/soc/motd.md is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/motd.md] at time 12:58:18.479468 (duration_in_ms=12.884)
[INFO    ] Running state [/opt/so/conf/soc/so-detections-backup.py] at time 12:58:18.479681
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/so-detections-backup.py]
[INFO    ] File /opt/so/conf/soc/so-detections-backup.py is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/so-detections-backup.py] at time 12:58:18.489598 (duration_in_ms=9.918)
[INFO    ] Running state [/usr/sbin/so-detections-runtime-status cron] at time 12:58:18.489831
[INFO    ] Executing state cron.present for [/usr/sbin/so-detections-runtime-status cron]
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Cron /usr/sbin/so-detections-runtime-status cron already present
[INFO    ] Completed state [/usr/sbin/so-detections-runtime-status cron] at time 12:58:18.494378 (duration_in_ms=4.548)
[INFO    ] Running state [python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detections-backup.log] a
 time 12:58:18.494593
[INFO    ] Executing state cron.present for [python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detec
ions-backup.log]
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Cron python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detections-backup.log already pres
nt
[INFO    ] Completed state [python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detections-backup.log]
at time 12:58:18.498102 (duration_in_ms=3.51)
[INFO    ] Running state [/opt/so/conf/soc/sigma_final_pipeline.yaml] at time 12:58:18.498315
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/sigma_final_pipeline.yaml]
[INFO    ] File /opt/so/conf/soc/sigma_final_pipeline.yaml is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/sigma_final_pipeline.yaml] at time 12:58:18.508049 (duration_in_ms=9.733)
[INFO    ] Running state [/opt/so/conf/soc/sigma_so_pipeline.yaml] at time 12:58:18.508263
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/sigma_so_pipeline.yaml]
[INFO    ] File /opt/so/conf/soc/sigma_so_pipeline.yaml is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/sigma_so_pipeline.yaml] at time 12:58:18.518205 (duration_in_ms=9.942)
[INFO    ] Running state [/opt/so/conf/soc/banner.md] at time 12:58:18.518440
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/banner.md]
[INFO    ] File /opt/so/conf/soc/banner.md is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/banner.md] at time 12:58:18.530139 (duration_in_ms=11.699)
[INFO    ] Running state [/usr/sbin] at time 12:58:18.530362
[INFO    ] Executing state file.recurse for [/usr/sbin]
[INFO    ] {'/usr/sbin': {'/usr/sbin': {'user': 939}, 'user': 939}}
[INFO    ] Loading fresh modules for state activity
[INFO    ] Completed state [/usr/sbin] at time 12:58:18.693156 (duration_in_ms=162.793)
[INFO    ] Running state [/opt/so/conf/soc/custom.js] at time 12:58:18.697686
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/custom.js]
[INFO    ] File /opt/so/conf/soc/custom.js is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/custom.js] at time 12:58:18.726344 (duration_in_ms=28.658)
[INFO    ] Running state [/opt/so/conf/soc/custom_roles] at time 12:58:18.726612
[INFO    ] Executing state file.managed for [/opt/so/conf/soc/custom_roles]
[INFO    ] File /opt/so/conf/soc/custom_roles is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/custom_roles] at time 12:58:18.740864 (duration_in_ms=14.252)
[INFO    ] Running state [/opt/so/conf/soc/soc_users_roles] at time 12:58:18.741285
[INFO    ] Executing state file.exists for [/opt/so/conf/soc/soc_users_roles]
[INFO    ] Path /opt/so/conf/soc/soc_users_roles exists
[INFO    ] Completed state [/opt/so/conf/soc/soc_users_roles] at time 12:58:18.742174 (duration_in_ms=0.888)
[INFO    ] Running state [/nsm/soc/uploads] at time 12:58:18.742374
[INFO    ] Executing state file.directory for [/nsm/soc/uploads]
[INFO    ] The directory /nsm/soc/uploads is in the correct state
[INFO    ] Completed state [/nsm/soc/uploads] at time 12:58:18.743618 (duration_in_ms=1.243)
[INFO    ] Running state [/opt/so/rules] at time 12:58:18.743817
[INFO    ] Executing state file.directory for [/opt/so/rules]
[INFO    ] The directory /opt/so/rules is in the correct state
[INFO    ] Completed state [/opt/so/rules] at time 12:58:18.745051 (duration_in_ms=1.234)
[INFO    ] Running state [/opt/so/conf/soc/ai_summary_repos] at time 12:58:18.745254
[INFO    ] Executing state file.directory for [/opt/so/conf/soc/ai_summary_repos]
[INFO    ] The directory /opt/so/conf/soc/ai_summary_repos is in the correct state
[INFO    ] Completed state [/opt/so/conf/soc/ai_summary_repos] at time 12:58:18.746474 (duration_in_ms=1.221)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:58:18.746674
[INFO    ] Executing state file.append for [/opt/so/conf/so-status/so-status.conf]
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop import
ng 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability laye
. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
[INFO    ] Executing command 'grep' in directory '/root'
[INFO    ] ['unless condition is true']
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:58:19.699940 (duration_in_ms=953.265)
[INFO    ] Running state [so-soc] at time 12:58:19.702328
[INFO    ] Executing state docker_container.running for [so-soc]
[INFO    ] {'container': {'Networks': {'bridge': {'IPConfiguration': {'old': 'automatic', 'new': 'not connected'}}, 'so
ridge': {'IPAMConfig': {'old': None, 'new': {'IPv4Address': '172.18.1.34'}}}}}, 'container_id': {'added': 'dfc6cf8f6fed
b65636cbc0d8c2652233205b76730f64838577264300afd6afb'}, 'state': {'old': None, 'new': 'running'}}
[INFO    ] Completed state [so-soc] at time 12:58:21.083678 (duration_in_ms=1381.35)
[INFO    ] Running state [/opt/so/conf/so-status/so-status.conf] at time 12:58:21.084101
[INFO    ] Executing state file.uncomment for [/opt/so/conf/so-status/so-status.conf]
[INFO    ] Pattern already uncommented
[INFO    ] Completed state [/opt/so/conf/so-status/so-status.conf] at time 12:58:21.087489 (duration_in_ms=3.388)
[INFO    ] Running state [/opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh &] at time 12:58:21.087733
[INFO    ] Executing state cron.present for [/opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh &]
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Cron /opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh & already present
[INFO    ] Completed state [/opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh &] at time 12:58:21.092478 (dura
ion_in_ms=4.746)
local:
----------
          ID: elastic_auth_pillar
    Function: file.managed
        Name: /opt/so/saltstack/local/pillar/elasticsearch/auth.sls
      Result: True
     Comment: File /opt/so/saltstack/local/pillar/elasticsearch/auth.sls is in the correct state
     Started: 12:58:08.187941
    Duration: 4984.637 ms
     Changes:
----------
          ID: kratosgroup
    Function: group.present
        Name: kratos
      Result: True
     Comment: Group kratos is present and up to date
     Started: 12:58:13.173952
    Duration: 5.298 ms
     Changes:
----------
          ID: kratos
    Function: user.present
      Result: True
     Comment: User kratos is present and up to date
     Started: 12:58:13.179982
    Duration: 12.661 ms
     Changes:
----------
          ID: kratosdir
    Function: file.directory
        Name: /nsm/kratos
      Result: True
     Comment: The directory /nsm/kratos is in the correct state
     Started: 12:58:13.192921
    Duration: 1.678 ms
     Changes:
----------
          ID: kratosdbdir
    Function: file.directory
        Name: /nsm/kratos/db
      Result: True
     Comment: The directory /nsm/kratos/db is in the correct state
     Started: 12:58:13.194809
    Duration: 1.272 ms
     Changes:
----------
          ID: kratoslogdir
    Function: file.directory
        Name: /opt/so/log/kratos
      Result: True
     Comment: The directory /opt/so/log/kratos is in the correct state
     Started: 12:58:13.196289
    Duration: 1.299 ms
     Changes:
----------
          ID: kratosschema
    Function: file.managed
        Name: /opt/so/conf/kratos/schema.json
      Result: True
     Comment: File /opt/so/conf/kratos/schema.json is in the correct state
     Started: 12:58:13.197874
    Duration: 12.663 ms
     Changes:
----------
          ID: kratosoidc
    Function: file.managed
        Name: /opt/so/conf/kratos/oidc.jsonnet
      Result: True
     Comment: File /opt/so/conf/kratos/oidc.jsonnet is in the correct state
     Started: 12:58:13.210800
    Duration: 11.895 ms
     Changes:
----------
          ID: kratosconfig
    Function: file.managed
        Name: /opt/so/conf/kratos/kratos.yaml
      Result: True
     Comment: File /opt/so/conf/kratos/kratos.yaml is in the correct state
     Started: 12:58:13.222966
    Duration: 22.463 ms
     Changes:
----------
          ID: append_so-kratos_so-status.conf
    Function: file.append
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: unless condition is true
     Started: 12:58:13.245678
    Duration: 1441.152 ms
     Changes:
----------
          ID: so-kratos
    Function: docker_container.running
      Result: True
     Comment: Container 'so-kratos' is already configured as specified
     Started: 12:58:14.688360
    Duration: 1367.533 ms
     Changes:
----------
          ID: delete_so-kratos_so-status.disabled
    Function: file.uncomment
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: Pattern already uncommented
     Started: 12:58:16.056164
    Duration: 2.502 ms
     Changes:
----------
          ID: wait_for_kratos
    Function: http.wait_for_successful_query
        Name: http://so-manager:4434/
      Result: True
     Comment: Status 404 was found.
                status: 404
                error: HTTP 404: Not Found
                body: 404 page not found

     Started: 12:58:16.059187
    Duration: 7.238 ms
     Changes:
----------
          ID: so-user.lock
    Function: file.missing
        Name: /var/tmp/so-user.lock
      Result: True
     Comment: Path /var/tmp/so-user.lock is missing
     Started: 12:58:16.066643
    Duration: 0.827 ms
     Changes:
----------
          ID: sync_es_users
    Function: cmd.run
        Name: so-user sync
      Result: True
     Comment: All files in creates exist
     Started: 12:58:16.071700
    Duration: 7.748 ms
     Changes:
----------
          ID: so-user_sync
    Function: cron.present
        Name: PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-user sync &>> /o
t/so/log/soc/sync.log
      Result: True
     Comment: Cron PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin /usr/sbin/so-user sync &
> /opt/so/log/soc/sync.log already present
     Started: 12:58:16.079945
    Duration: 15.084 ms
     Changes:
----------
          ID: sigmarepodir
    Function: file.directory
        Name: /opt/so/conf/sigma/repos
      Result: True
     Comment: The directory /opt/so/conf/sigma/repos is in the correct state
     Started: 12:58:16.095272
    Duration: 1.508 ms
     Changes:
----------
          ID: socdirelastaertrules
    Function: file.directory
        Name: /opt/so/rules/elastalert/rules
      Result: True
     Comment: The directory /opt/so/rules/elastalert/rules is in the correct state
     Started: 12:58:16.097009
    Duration: 1.323 ms
     Changes:
----------
          ID: socdir
    Function: file.directory
        Name: /opt/so/conf/soc/fingerprints
      Result: True
     Comment: The directory /opt/so/conf/soc/fingerprints is in the correct state
     Started: 12:58:16.098546
    Duration: 1.262 ms
     Changes:
----------
          ID: socdatadir
    Function: file.directory
        Name: /nsm/soc/jobs
      Result: True
     Comment: The directory /nsm/soc/jobs is in the correct state
     Started: 12:58:16.100026
    Duration: 1.346 ms
     Changes:
----------
          ID: soclogdir
    Function: file.directory
        Name: /opt/so/log/soc
      Result: True
     Comment: The directory /opt/so/log/soc is in the correct state
     Started: 12:58:16.101600
    Duration: 5.851 ms
     Changes:
----------
          ID: socsaltdir
    Function: file.directory
        Name: /opt/so/conf/soc/queue
      Result: True
     Comment: The directory /opt/so/conf/soc/queue is in the correct state
     Started: 12:58:16.107673
    Duration: 1.264 ms
     Changes:
----------
          ID: socanalytics
    Function: file.managed
        Name: /opt/so/conf/soc/analytics.js
      Result: True
     Comment: File /opt/so/conf/soc/analytics.js is in the correct state
     Started: 12:58:16.109165
    Duration: 13.519 ms
     Changes:
----------
          ID: socconfig
    Function: file.managed
        Name: /opt/so/conf/soc/soc.json
      Result: True
     Comment: File /opt/so/conf/soc/soc.json is in the correct state
     Started: 12:58:16.122919
    Duration: 2343.268 ms
     Changes:
----------
          ID: socmotd
    Function: file.managed
        Name: /opt/so/conf/soc/motd.md
      Result: True
     Comment: File /opt/so/conf/soc/motd.md is in the correct state
     Started: 12:58:18.466584
    Duration: 12.884 ms
     Changes:
----------
          ID: filedetectionsbackup
    Function: file.managed
        Name: /opt/so/conf/soc/so-detections-backup.py
      Result: True
     Comment: File /opt/so/conf/soc/so-detections-backup.py is in the correct state
     Started: 12:58:18.479680
    Duration: 9.918 ms
     Changes:
----------
          ID: crondetectionsruntime
    Function: cron.present
        Name: /usr/sbin/so-detections-runtime-status cron
      Result: True
     Comment: Cron /usr/sbin/so-detections-runtime-status cron already present
     Started: 12:58:18.489830
    Duration: 4.548 ms
     Changes:
----------
          ID: crondetectionsbackup
    Function: cron.present
        Name: python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detections-backup.log
      Result: True
     Comment: Cron python3 /opt/so/conf/soc/so-detections-backup.py &>> /opt/so/log/soc/detections-backup.log already p
esent
     Started: 12:58:18.494592
    Duration: 3.51 ms
     Changes:
----------
          ID: socsigmafinalpipeline
    Function: file.managed
        Name: /opt/so/conf/soc/sigma_final_pipeline.yaml
      Result: True
     Comment: File /opt/so/conf/soc/sigma_final_pipeline.yaml is in the correct state
     Started: 12:58:18.498316
    Duration: 9.733 ms
     Changes:
----------
          ID: socsigmasopipeline
    Function: file.managed
        Name: /opt/so/conf/soc/sigma_so_pipeline.yaml
      Result: True
     Comment: File /opt/so/conf/soc/sigma_so_pipeline.yaml is in the correct state
     Started: 12:58:18.508263
    Duration: 9.942 ms
     Changes:
----------
          ID: socbanner
    Function: file.managed
        Name: /opt/so/conf/soc/banner.md
      Result: True
     Comment: File /opt/so/conf/soc/banner.md is in the correct state
     Started: 12:58:18.518440
    Duration: 11.699 ms
     Changes:
----------
          ID: soc_sbin
    Function: file.recurse
        Name: /usr/sbin
      Result: True
     Comment: Recursively updated /usr/sbin
     Started: 12:58:18.530363
    Duration: 162.793 ms
     Changes:
              ----------
              /usr/sbin:
                  ----------
                  /usr/sbin:
                      ----------
                      user:
                          939
                  user:
                      939
----------
          ID: soccustom
    Function: file.managed
        Name: /opt/so/conf/soc/custom.js
      Result: True
     Comment: File /opt/so/conf/soc/custom.js is in the correct state
     Started: 12:58:18.697686
    Duration: 28.658 ms
     Changes:
----------
          ID: soccustomroles
    Function: file.managed
        Name: /opt/so/conf/soc/custom_roles
      Result: True
     Comment: File /opt/so/conf/soc/custom_roles is in the correct state
     Started: 12:58:18.726612
    Duration: 14.252 ms
     Changes:
----------
          ID: socusersroles
    Function: file.exists
        Name: /opt/so/conf/soc/soc_users_roles
      Result: True
     Comment: Path /opt/so/conf/soc/soc_users_roles exists
     Started: 12:58:18.741286
    Duration: 0.888 ms
     Changes:
----------
          ID: socuploaddir
    Function: file.directory
        Name: /nsm/soc/uploads
      Result: True
     Comment: The directory /nsm/soc/uploads is in the correct state
     Started: 12:58:18.742375
    Duration: 1.243 ms
     Changes:
----------
          ID: socsigmarepo
    Function: file.directory
        Name: /opt/so/rules
      Result: True
     Comment: The directory /opt/so/rules is in the correct state
     Started: 12:58:18.743817
    Duration: 1.234 ms
     Changes:
----------
          ID: socsensoronirepos
    Function: file.directory
        Name: /opt/so/conf/soc/ai_summary_repos
      Result: True
     Comment: The directory /opt/so/conf/soc/ai_summary_repos is in the correct state
     Started: 12:58:18.745253
    Duration: 1.221 ms
     Changes:
----------
          ID: append_so-soc_so-status.conf
    Function: file.append
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: unless condition is true
     Started: 12:58:18.746675
    Duration: 953.265 ms
     Changes:
----------
          ID: so-soc
    Function: docker_container.running
      Result: True
     Comment: Created container 'so-soc'
     Started: 12:58:19.702328
    Duration: 1381.35 ms
     Changes:
              ----------
              container:
                  ----------
                  Networks:
                      ----------
                      bridge:
                          ----------
                          IPConfiguration:
                              ----------
                              new:
                                  not connected
                              old:
                                  automatic
                      sobridge:
                          ----------
                          IPAMConfig:
                              ----------
                              new:
                                  ----------
                                  IPv4Address:
                                      172.18.1.34
                              old:
                                  None
              container_id:
                  ----------
                  added:
                      dfc6cf8f6fed9b65636cbc0d8c2652233205b76730f64838577264300afd6afb
              state:
                  ----------
                  new:
                      running
                  old:
                      None
----------
          ID: delete_so-soc_so-status.disabled
    Function: file.uncomment
        Name: /opt/so/conf/so-status/so-status.conf
      Result: True
     Comment: Pattern already uncommented
     Started: 12:58:21.084101
    Duration: 3.388 ms
     Changes:
----------
          ID: salt-relay
    Function: cron.present
        Name: /opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh &
      Result: True
     Comment: Cron /opt/so/saltstack/default/salt/soc/files/bin/salt-relay.sh & already present
     Started: 12:58:21.087732
    Duration: 4.746 ms
     Changes:

Summary for local
-------------
Succeeded: 42 (changed=2)
Failed:     0
-------------
Total states run:     42
Total run time:   12.881 s
[root@so-manager soc]#

[reyesj2]

Last month, I attempted to change SO to direct (NAT) internet communications

Can you run salt-call pillar.get manager? If you removed proxy configuration your output should look like

local:
    ----------
    elastalert:
        1
    no_proxy:
    proxy:

[Gregg3]

Here is what I get. Does this indicate a problem that I can fix?

[root@so-manager kratos]# salt-call pillar.get manager
local:
    ----------
    elastalert:
        1
    no_proxy:
        1
    playbook:
        1
    proxy:
        0

[reyesj2]

Can you manually edit the file /opt/so/saltstack/local/pillar/manager/soc_manager.sls
The file should look like

manager:
  proxy:  ''
  no_proxy: ''
  elastalert: 1
  playbook: 1

Then run sudo so-restart soc

Did you manually edit this file before or did you make your changes via the SOC UI?

[Gregg3]

THANK YOU!

I don't recall editing that file. I tried making changes via SOC UI, but even after reboots, Security Onion was still using proxy for some things. I eventually got desperate and made changes anywhere I saw the proxy in a configuration file. So it's quite likely that I did edit this file and didn't follow the right syntax.