soup reports 2 Fails of 777 2.3.100 -> 2.3.110-20220407 hotfix

[LWHaysUSDC]

Hi all,

I had been running SO 2.3.100 on a centos-7 based grid (5 sensors, 1 search, 1 manager) and I rebooted first (by request at the CLI) and then made soup tonight, so I believe I got 2/3/110-20220407.

soup of soup went fine, "run again"

2nd run ended with:

Summary for local
--------------
Succeeded: 775 (changed=26)
Failed:      2
--------------
Total states run:     777
Total run time:   106.415 s
Soup failed with error 1: Unhandled error
Unhandled error occured, please check /root/soup.log for details.

Starting crond service at 00:13:45.856978
Successfully started crond.

Enabling highstate.
local:
    ----------
    msg:
        Info: highstate state already enabled.
    res:
        True

I reviewed soup.log, but I'm naive and don't know what I'm looking for. (soup has always just worked...)

I did run soup again (I trust Doug and the team, a lot), but it just said
"You are already running the latest version of Security Onion."

There is the following about 1100 lines upstream, which seems relevant (sorry for the big blockquote, as I said, I don't know what I'm looking for....):

----------
          ID: load_elastic_pipelines
    Function: cmd.run
        Name: /usr/sbin/so-filebeat-module-setup
      Result: True
     Comment: Command "/usr/sbin/so-filebeat-module-setup" run
     Started: 00:07:58.030540
    Duration: 183221.578 ms
     Changes:   
              ----------
              pid:
                  167389
              retcode:
                  0
              stderr:
              stdout:
                  Waiting for ElasticSearch...connected!
                  Testing to see if the pipelines are already applied
                  Setting up ingest pipeline(s)
                  Loading activemq
                  Loaded Ingest pipelines
                  Loading apache
                  Loaded Ingest pipelines
                  Loading auditd
                  Loaded Ingest pipelines
                  Loading aws
                  Loaded Ingest pipelines
                  Loading azure
                  Loaded Ingest pipelines
                  Loading barracuda
                  Loaded Ingest pipelines
                  Loading bluecoat
                  Loaded Ingest pipelines
                  Loading cef
                  Loaded Ingest pipelines
                  Loading checkpoint
                  Loaded Ingest pipelines
                  Loading cisco
                  Loaded Ingest pipelines
                  Loading coredns
                  Loaded Ingest pipelines
                  Loading crowdstrike
                  Loaded Ingest pipelines
                  Loading cyberark
                  Loaded Ingest pipelines
                  Loading cylance
                  Loaded Ingest pipelines
                  Loading elasticsearch
                  Loaded Ingest pipelines
                  Loading envoyproxy
                  Loaded Ingest pipelines
                  Loading f5
                  Loaded Ingest pipelines
                  Loading fortinet
                  Loaded Ingest pipelines
                  Loading gcp
                  Loaded Ingest pipelines
                  Loading google_workspace
                  Loaded Ingest pipelines
                  Loading googlecloud
                  Loaded Ingest pipelines
                  Loading gsuite
                  Loaded Ingest pipelines
                  Loading haproxy
                  Loaded Ingest pipelines
                  Loading ibmmq
                  Loaded Ingest pipelines
                  Loading icinga
                  Loaded Ingest pipelines
                  Loading iis
                  Loaded Ingest pipelines
                  Loading imperva
                  Loaded Ingest pipelines
                  Loading infoblox
                  Loaded Ingest pipelines
                  Loading iptables
                  Loaded Ingest pipelines
                  Loading juniper
                  Loaded Ingest pipelines
                  Loading kafka
                  Loaded Ingest pipelines
                  Loading kibana
                  Loaded Ingest pipelines
                  Loading logstash
                  Loaded Ingest pipelines
                  Loading microsoft
                  Loaded Ingest pipelines
                  Loading mongodb
                  Loaded Ingest pipelines
                  Loading mssql
                  Loaded Ingest pipelines
                  Loading mysql
                  Loaded Ingest pipelines
                  Loading nats
                  Loaded Ingest pipelines
                  Loading netscout
                  Loaded Ingest pipelines
                  Loading nginx
                  Loaded Ingest pipelines
                  Loading o365
                  Loaded Ingest pipelines
                  Loading okta
                  Loaded Ingest pipelines
                  Loading osquery
                  Loaded Ingest pipelines
                  Loading panw
                  Loaded Ingest pipelines
                  Loading postgresql
                  Loaded Ingest pipelines
                  Loading rabbitmq
                  Loaded Ingest pipelines
                  Loading radware
                  Loaded Ingest pipelines
                  Loading redis
                  Loaded Ingest pipelines
                  Loading santa
                  Loaded Ingest pipelines
                  Loading snort
                  Loaded Ingest pipelines
                  Loading snyk
                  Loaded Ingest pipelines
                  Loading sonicwall
                  Loaded Ingest pipelines
                  Loading sophos
                  Loaded Ingest pipelines
                  Loading squid
                  Loaded Ingest pipelines
                  Loading suricata
                  Loaded Ingest pipelines
                  Loading system
                  Loaded Ingest pipelines
                  Loading threatintel
                  Loaded Ingest pipelines
                  Loading tomcat
                  Loaded Ingest pipelines
                  Loading traefik
                  Loaded Ingest pipelines
                  Loading zeek
                  Loaded Ingest pipelines
                  Loading zscaler
                  Loaded Ingest pipelines
----------
          ID: lasthighstate
    Function: file.touch
        Name: /opt/so/log/salt/lasthighstate
      Result: True
     Comment: Updated times on file /opt/so/log/salt/lasthighstate
     Started: 00:11:01.253101
    Duration: 5.635 ms
     Changes:   
              ----------
              touched:
                  /opt/so/log/salt/lasthighstate





Summary for local
--------------
Succeeded: 775 (changed=125)
Failed:      2
--------------
Total states run:     777
Total run time:   735.174 s

Killing all Salt jobs across the grid.
ERROR: Minions returned with non-zero exit code
somanager_manager:
sosearch01_searchnode:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902
soabq_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902
sosf_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902
sogld_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902
soro_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902
solc_sensor:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
    
    salt-run jobs.lookup_jid 20220409001105008902

Killing any queued Salt jobs on the manager.

Storing salt-master pid.
Found salt-master PID 136294

sudo salt-run jobs.lookup_jid 20220409001105008902 returns
somanager_manager:
[INFO ] Runner completed: 20220409020659178949

Which seems reassuring.

There's about 10K lines above that until we have the previous "Summary for local" (Succeeded: 1 (changed=1), Failed: 0 FWIW)
I don't see anything that stands out to me in that except a big block of non-human-formatted text that begins:

----------
          ID: so-kibana-dashboard-load
    Function: cmd.run
        Name: /usr/sbin/so-kibana-config-load -i /opt/so/conf/kibana/saved_objects.ndjson.template
      Result: True
     Comment: Command "/usr/sbin/so-kibana-config-load -i /opt/so/conf/kibana/saved_objects.ndjson.template" run
     Started: 00:06:09.659520
    Duration: 3095.999 ms
     Changes:   
              ----------
              pid:
                  163290
              retcode:
                  0
              stderr:
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                   Dload  Upload   Total   Spent    Left  Speed
                  
                    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
                  100 3220k    0     0  100 3220k      0  3197k  0:00:01  0:00:01 --:--:-- 3200k
                  100 3341k  100  121k  100 3220k  84691  2191k  0:00:01  0:00:01 --:--:-- 2193k
              stdout:
                  Waiting for value 'Elastic' at 'http://localhost:5601/app/kibana' (1/300)
                  Received expected response; proceeding.
                  {"successCount":823,"success":true,"warnings":[],"successResults":[{"type":"index-pattern","id":"*:elastalert_status*","meta":{"title":"*:elastalert_status*","icon":"indexP
atternApp"},"overwrite":true},{"type":"index-pattern","id":"*:logstash-*","meta":{"title":"*:logstash-*","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"7f822930-6e
a4-11ea-9266-1fd14ca6af34","meta":{"title":"Security Onion - Network Data","icon":"visualizeA......

FWIW, I can log into my manager node, there are alerts there for the last 24 hours, My Grafana tab has graphs, etc and sudo salt * so.status returns all green for all nodes.

Am I just triggered unnecessarily? What else should I be looking at? Thanks.

Larry

[Acewiza]

I think we are all waiting to find out what the issue was with that last hotfix.

[TOoSmOotH]

It looks like the agents were probably upgrading at the time that soup ran and were unable to talk to the salt master service. The verification at the end is just looking for the word "ERROR" even though some errors are not harmful. If all of your nodes are green and data is coming in then you are where you want to be. Salt will ensure the box is in the right "state" even if it takes multiple runs to get there.

@Acewiza As far as what the hotfix addressed was specific to Ubuntu. We did update the version of salt for centos as well since it was related to a critical CVE. We did introduce a regression in soup for airgap where we were telling soup to update salt before the new repo files were copied to the airgap repo located on the manager. The core issue here was Saltstack pulled the salt package from their repo causing all new Ubuntu installs to fail. We had to release multiple hotfixes as we got different scenarios from our customers and community. To prevent this from happening again, we are now hosting a copy of the Ubuntu salt repo at the securityonion public repo.

Normally we would not do a hotfix for salt and wait to upgrade in our next major release but we couldn't leave our Ubuntu users in a broken state. None of the containers or the actual states changed in the hotfix, just the delivery mechanism for the Ubuntu packages.

[Acewiza]

Understand. A Little. ;-) I usually wait to see who else is complaining about something before doing too much investigation. Helps with my short attention span...

[LWHaysUSDC]

Thank You @TOoSmOotH. Much appreciated (@Acewiza too)