Critical Error - Verifying Setup #1636

secraven · 2020-10-23T20:04:55Z

secraven
Oct 23, 2020

I've tried installing SO2 a number of times. My platform is ESXi. This the first critical error. I've enabled 8 CPU's, 32 GB RAM, 1TB Disk space. I've also tried disabling thehive and playbook, but continue to get errors. Looking forward to a successful install.

[CRITICAL] Rendering SLS 'base:ssl' failed: mapping values are not allowed in this context

`Enabling checkin at boot

95% - VERIFYING SETUP

[INFO ] Loading fresh modules for state activity
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[INFO ] Syncing clouds for environment 'base'
[INFO ] Loading cache from salt://_clouds, for base
[INFO ] Caching directory '_clouds/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[INFO ] Syncing beacons for environment 'base'
[INFO ] Loading cache from salt://_beacons, for base
[INFO ] Caching directory '_beacons/' for environment 'base'
[INFO ] Fetching file from saltenv 'base', done '_beacons/zeek.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_beacons/zeek.py' to '/var/cache/salt/minion/extmods/beacons/zeek.py'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[INFO ] Syncing modules for environment 'base'
[INFO ] Loading cache from salt://_modules, for base
[INFO ] Caching directory '_modules/' for environment 'base'
[INFO ] Fetching file from saltenv 'base', done '_modules/healthcheck.py'
[INFO ] Fetching file from saltenv 'base', done '_modules/needs_restarting.py'
[INFO ] Fetching file from saltenv 'base', done '_modules/so.py'
[INFO ] Fetching file from saltenv 'base', done '_modules/telegraf.py'
[INFO ] Fetching file from saltenv 'base', done '_modules/zeekctl.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_modules/healthcheck.py' to '/var/cache/salt/minion/extmods/modules/healthcheck.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_modules/needs_restarting.py' to '/var/cache/salt/minion/extmods/modules/needs_restarting.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_modules/so.py' to '/var/cache/salt/minion/extmods/modules/so.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_modules/telegraf.py' to '/var/cache/salt/minion/extmods/modules/telegraf.py'
[INFO ] Copying '/var/cache/salt/minion/files/base/_modules/zeekctl.py' to '/var/cache/salt/minion/extmods/modules/zeekctl.py'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/states'
[INFO ] Syncing states for environment 'base'
[INFO ] Loading cache from salt://_states, for base
[INFO ] Caching directory '_states/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[INFO ] Syncing sdb for environment 'base'
[INFO ] Loading cache from salt://_sdb, for base
[INFO ] Caching directory '_sdb/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[INFO ] Syncing renderers for environment 'base'
[INFO ] Loading cache from salt://_renderers, for base
[INFO ] Caching directory '_renderers/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[INFO ] Syncing returners for environment 'base'
[INFO ] Loading cache from salt://_returners, for base
[INFO ] Caching directory '_returners/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/output'
[INFO ] Syncing output for environment 'base'
[INFO ] Loading cache from salt://_output, for base
[INFO ] Caching directory '_output/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[INFO ] Syncing utils for environment 'base'
[INFO ] Loading cache from salt://_utils, for base
[INFO ] Caching directory '_utils/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[INFO ] Syncing log_handlers for environment 'base'
[INFO ] Loading cache from salt://_log_handlers, for base
[INFO ] Caching directory '_log_handlers/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[INFO ] Syncing executors for environment 'base'
[INFO ] Loading cache from salt://_executors, for base
[INFO ] Caching directory '_executors/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[INFO ] Syncing proxy for environment 'base'
[INFO ] Loading cache from salt://_proxy, for base
[INFO ] Caching directory '_proxy/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[INFO ] Syncing engines for environment 'base'
[INFO ] Loading cache from salt://_engines, for base
[INFO ] Caching directory '_engines/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[INFO ] Syncing thorium for environment 'base'
[INFO ] Loading cache from salt://_thorium, for base
[INFO ] Caching directory '_thorium/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[INFO ] Syncing serializers for environment 'base'
[INFO ] Loading cache from salt://_serializers, for base
[INFO ] Caching directory '_serializers/' for environment 'base'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[INFO ] Syncing matchers for environment 'base'
[INFO ] Loading cache from salt://_matchers, for base
[INFO ] Caching directory '_matchers/' for environment 'base'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'yum/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'salt/minion.sls'
[INFO ] Fetching file from saltenv 'base', done 'salt/map.jinja'
[INFO ] Fetching file from saltenv 'base', done 'salt/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'patch/os/schedule.sls'
[INFO ] Fetching file from saltenv 'base', done 'motd/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'salt/master.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[CRITICAL] Rendering SLS 'base:ssl' failed: mapping values are not allowed in this context
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'telegraf/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'influxdb/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'grafana/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'healthcheck/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'redis/init.sls'
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[INFO ] Loading fresh modules for state activity
[CRITICAL] Rendering SLS 'base:elastalert' failed: while parsing a block mapping
did not find expected key
in "", line 117, column 9
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', done 'schedule.sls'
[INFO ] Loading fresh modules for state activity
[CRITICAL] Rendering SLS 'base:soctopus' failed: while parsing a block mapping
did not find expected key
in "", line 72, column 9
[INFO ] Fetching file from saltenv 'base', done 'docker_clean/init.sls'
local:
Data failed to compile:

Rendering SLS 'base:ssl' failed: mapping values are not allowed in this context

Rendering SLS 'base:elastalert' failed: while parsing a block mapping

did not find expected key
in "", line 117, column 9

Rendering SLS 'base:soctopus' failed: while parsing a block mapping

did not find expected key
in "", line 72, column 9
Errors detected during setup; skipping post-setup steps to allow for analysis of failures.
Installer removing the following files:
/root/installtmp:
total 0
drwxr-xr-x. 3 root root 21 Oct 23 19:08 pillar

/root/installtmp/pillar:
total 0
drwxr-xr-x. 2 root root 42 Oct 23 19:10 minions

/root/installtmp/pillar/minions:
total 4
-rw-r--r--. 1 root root 963 Oct 23 19:12 securityonion_standalone.sls

Stopping mysql...

Error response from daemon: No such container: so-mysql
Error: No such container: so-mysql`

TOoSmOotH · 2020-10-23T21:05:34Z

TOoSmOotH
Oct 23, 2020
Maintainer

Do you have any special characters in your hostname?

3 replies

secraven Oct 23, 2020
Author

I creatively chose "securityonion" as my host name. No special characters or punctuation involved.

secraven Oct 24, 2020
Author

I tried another install and chose to only install Grafana & Strelka. The install failed again at Verifying Setup. Here is a snip of the logs:

Summary for local
--------------
Succeeded: 422 (changed=48)
Failed:      0
--------------
Total states run:     422
Total run time:   262.870 s
Errors detected during setup; skipping post-setup steps to allow for analysis of failures.
Installer removing the following files:
/root/installtmp:
total 0
drwxr-xr-x. 3 root root 21 Oct 24 21:03 pillar

/root/installtmp/pillar:
total 0
drwxr-xr-x. 2 root root 33 Oct 24 21:08 minions

/root/installtmp/pillar/minions:
total 4
-rw-r--r--. 1 root root 999 Oct 24 21:09 siem_standalone.sls
=========================================================================
Stopping mysql...
=========================================================================
Error response from daemon: No such container: so-mysql
Error: No such container: so-mysql

TOoSmOotH Oct 24, 2020
Maintainer

reboot and run so-status.

secraven · 2020-10-25T19:22:08Z

secraven
Oct 25, 2020
Author

This is the output from so-status:

Checking Docker status

    Docker --------------------------------------------------------------------------------------------------------------------------------------- [ OK ]   

Checking container statuses

    so-aptcacherng ------------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-curator -------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-dockerregistry ---------------------------------------------------------------------------------------------------------------------------- [ OK ]   
    so-elastalert ----------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-elasticsearch -------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-filebeat ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-grafana -------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-idstools ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-influxdb ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-kibana --------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-kratos --------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-logstash ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-nginx ---------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-redis ---------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]   
    so-sensoroni ------------------------------------------------------------------------------------------------------------------------------ [ ERROR ]

    so-soc ------------------------------------------------------------------------------------------------------------------------------------ [ ERROR ]

    so-soctopus ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-steno ---------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-strelka-coordinator -------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-strelka-filestream --------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-strelka-frontend ----------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-strelka-gatekeeper --------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-strelka-manager ------------------------------------------------------------------------------------------------------------------------ [ ERROR ]

    so-suricata ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-telegraf ------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]

    so-zeek ----------------------------------------------------------------------------------------------------------------------------------- [ ERROR ]

5 replies

secraven Oct 26, 2020
Author

Any suggestions? I am looking forward to giving SO2 a spin!

ghost Oct 27, 2020

I'm just a rando SO user myself, I've got nothing to to with the project. But I too am using ESXi on an ancient, creaky PowerEdge. After boot, how long before you do the so-status query? I ask because in my case with my old box, it takes sometimes upwards of 6 or 7 minutes before all those "Errors" turn to "OK". I actually now wait a bit before even initiating so-allows or soup, just so all the containers have time to get spun up.

That may not be the case with you, but I figured I should mention it. Also, I don't get the same errors as you posted in the OP, so I may be speaking out of my bleep here...

secraven Oct 27, 2020
Author

Thanks for your input. I ran it again and everything is "ok" and green. When I go to the main SO page, I get the following error in red for Alerts & Hunt. I also don't see any data in ELK.

search_phase_execution_exception: all shards failed -> { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Text fields are not optimised for...

secraven Oct 27, 2020
Author

I sure hope ESXi is still a viable option for SO. I cannot justify running a rack mount server in my home office. I am hoping to get this up and running but I keep running into show stoppers.

weslambert Oct 27, 2020

Have you tried verifying that you are actually receiving data on the wire using tcpdump or something similar?

What is the output of the following?

curl -s http://localhost:9200/_cat/indices

Have you tried checking for clues in /opt/so/log/ ?

secraven · 2020-10-27T21:59:53Z

secraven
Oct 27, 2020
Author

Wes -

Thank you for your response.

Yes tcpdump is seeing a ton of traffic on the monitoring interface.

Here is the output of the curl request:

curl -s http://localhost:9200/_cat/indices
green open so-ids-2020.10.24         _aj1GEFGQ5ioR4xItlPEsQ 1 0    760 0     2mb     2mb
green open elastalert_status_status  GsuQSt2CS_KQWp91lNTXog 1 0      0 0    208b    208b
green open so-ids-2020.10.26         -PR0fTd7RwuLTq0UtiXVIQ 1 0   8825 0  19.9mb  19.9mb
green open elastalert_status         wgiZmGSgTRy6nF_UucthIw 1 0      0 0    208b    208b
green open elastalert_status_past    X-yvWHbPTW6nCjwCL1whPA 1 0      0 0    208b    208b
green open so-ids-2020.10.25         8WjIauYNS0i3stD6uRKwvQ 1 0   5956 0  10.4mb  10.4mb
green open so-zeek-2020.10.26        R8wEXa2lQKuFCow8fILDnw 5 0 632583 0     1gb     1gb
green open so-ids-2020.10.27         QzMYbjrGQyyGS0pbxqEl9g 1 0   8215 0  19.1mb  19.1mb
green open so-zeek-2020.10.27        vwCqdIhXT3eAR-Jya2_Sqw 5 0 514777 0 867.4mb 867.4mb
green open elastalert_status_silence MYjxPn0xScuoF45at9-KDA 1 0      0 0    208b    208b
green open .kibana_1                 p9ymh319Qf6IMBcl4sOwsA 1 0     14 3  34.5kb  34.5kb
green open so-zeek-2020.10.24        lRPmUqz3RXKa3vW57g7Gkw 5 0  57607 0  97.1mb  97.1mb
green open so-zeek-2020.10.25        8sfrzk6dTk27k-HZwrHONA 5 0 409217 0 636.6mb 636.6mb
green open elastalert_status_error   ZfF9N79fR5u_Rk8CT4Ikcw 1 0      0 0    208b    208b
green open so-strelka-2020.10.25     cV0mbkZYQ3e3c6PPld-YmA 1 0    156 0     1mb     1mb
green open so-strelka-2020.10.26     nGDEw210S-m4Ng4cn_5uYw 1 0    246 0 887.8kb 887.8kb

I found this error message in the logstash section.

[2020-10-27T00:02:11,800][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
Filebeat:
2020-10-27T00:38:50.981Z WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

NGINX

2020/10/24 21:31:11 [error] 6#6: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.17.0.1, server: _, request: "POST /sensoroniagents/api/sensor HTTP/1.1", subrequest: "/50x.html", upstream: "http://192.168.1.10:9822/50x.html", host: "siem"
2020/10/24 21:31:11 [error] 6#6: *5 auth request unexpected status: 502 while sending to client, client: 172.17.0.1, server: _, request: "POST /sensoroniagents/api/sensor HTTP/1.1", upstream: "http://192.168.1.10:9822/api/sensor", host: "siem"

1 reply

secraven Oct 28, 2020
Author

Do I need to expand my so-allow networks? I've only allowed my home network subnet for an analyst.

Critical Error - Verifying Setup #1636

Uh oh!

secraven Oct 23, 2020

`Enabling checkin at boot

95% - VERIFYING SETUP

did not find expected key in "", line 117, column 9

/root/installtmp/pillar/minions: total 4 -rw-r--r--. 1 root root 963 Oct 23 19:12 securityonion_standalone.sls

Stopping mysql...

Replies: 3 comments · 9 replies

Uh oh!

TOoSmOotH Oct 23, 2020 Maintainer

Uh oh!

secraven Oct 23, 2020 Author

Uh oh!

secraven Oct 24, 2020 Author

Uh oh!

TOoSmOotH Oct 24, 2020 Maintainer

Uh oh!

Uh oh!

secraven Oct 25, 2020 Author

Uh oh!

secraven Oct 26, 2020 Author

Uh oh!

ghost Oct 27, 2020

Uh oh!

secraven Oct 27, 2020 Author

Uh oh!

secraven Oct 27, 2020 Author

Uh oh!

weslambert Oct 27, 2020

Uh oh!

Uh oh!

secraven Oct 27, 2020 Author

Uh oh!

secraven Oct 28, 2020 Author

secraven
Oct 23, 2020

did not find expected key
in "", line 117, column 9

/root/installtmp/pillar/minions:
total 4
-rw-r--r--. 1 root root 963 Oct 23 19:12 securityonion_standalone.sls

Replies: 3 comments 9 replies

TOoSmOotH
Oct 23, 2020
Maintainer

secraven Oct 23, 2020
Author

secraven Oct 24, 2020
Author

TOoSmOotH Oct 24, 2020
Maintainer

secraven
Oct 25, 2020
Author

secraven Oct 26, 2020
Author

secraven Oct 27, 2020
Author

secraven Oct 27, 2020
Author

secraven
Oct 27, 2020
Author

secraven Oct 28, 2020
Author