wait_start and XX_state_not_allowed after SOUP

[bretthanus]

we have a distributed setup - manager node, 3 search nodes and 6 sensors- SOUP moved everything up to 2.3.130, except 3 sensors successfully- the 3 sensors stuck on 2.3.120 show faulted in the grid, and so-status on each shows 'wait_start'

if i try to manually start with so-start filebeat for example, i get the 'test.fail_without_changes' failure filebeat_state_not_allowed.

I figured the quickest way to resolve would be to simply reinstall the sensor- so i git cloned, and re-ran the so-network-setup, seems to do everything successfully, sosetup.log shows this as last line:
Errors detected during setup; skipping post-setup steps to allow for analysis of failures.
checking the error.log its empty... grepping for 'error' in sosetup.log provides:
` grep -i error -B12 sosetup.log

Succeeded: 31 (changed=7)
Failed: 0

Total states run: 31
Total run time: 10.881 s
Applying SSL state
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'ssl/init.sls'
[INFO ] Fetching file from saltenv 'base', ** done ** 'allowed_states.map.jinja'
[INFO ] Running state [ssl_state_not_allowed] at time 12:13:14.037699
[INFO ] Executing state test.fail_without_changes for [ssl_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 0
Failed: 1

Total states run: 1
Total run time: 4.413 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'firewall/init.sls'
[INFO ] Running state [firewall_state_not_allowed] at time 12:13:22.225883
[INFO ] Executing state test.fail_without_changes for [firewall_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 2
Failed: 0

Total states run: 2
Total run time: 852.422 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'common/init.sls'
[INFO ] Running state [common_state_not_allowed] at time 12:13:41.598584
[INFO ] Executing state test.fail_without_changes for [common_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 0
Failed: 1

Total states run: 1
Total run time: 4.330 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'nginx/init.sls'
[INFO ] Running state [nginx_state_not_allowed] at time 12:13:50.013547
[INFO ] Executing state test.fail_without_changes for [nginx_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 0
Failed: 1

Total states run: 1
Total run time: 7.355 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'pcap/init.sls'
[INFO ] Running state [pcap_state_not_allowed] at time 12:13:58.383198
[INFO ] Executing state test.fail_without_changes for [pcap_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 0
Failed: 1

Total states run: 1
Total run time: 4.650 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'suricata/init.sls'
[INFO ] Running state [suricata_state_not_allowed] at time 12:14:07.428474
[INFO ] Executing state test.fail_without_changes for [suricata_state_not_allowed]
[ERROR ] Failure!

Summary for local

Succeeded: 0
Failed: 1

Total states run: 1
Total run time: 4.644 ms
[INFO ] Loading fresh modules for state activity
[INFO ] Fetching file from saltenv 'base', ** done ** 'zeek/init.sls'
[INFO ] Running state [zeek_state_not_allowed] at time 12:14:22.625794
[INFO ] Executing state test.fail_without_changes for [zeek_state_not_allowed]
[ERROR ] Failure!

`
which seem to be the same salt errors if attempting to manually start the containers.

6 sensors, 3 are good..

please point me in a direction -

thanks

[bretthanus]

I executed the steps to upgrade from 130 to 140, and the 2 of the 3 sensors came back online- the 1 that didnt required me to retry the failed install.
seems to work again...