soup upgrade from 2.3.240 to 250, elastic restart issues

[The1Waterman]

After a soup upgrade on a standalone system from 2.3.240 to 250, I get an error on all alerts/dashboards/hunt/cases etc,

"Request failed with status code 500
See the Help section of the Security Onion documentation for additional troubleshooting guidance. "

re running soup the process hangs with the following:

This version of the soup script is up to date. Proceeding.
Checking for unsupported Elasticsearch indices...
No unsupported indices found.
Checking Elastalert indices for compatibility...
Waiting for ElasticSearch...connected!

Stopping elastalert...

Error response from daemon: No such container: so-elastalert
Error: No such container: so-elastalert
Deleting Elastalert indices to prevent issues with upgrade to Elastic 8...
.....................................................

eventually stopping with:

running so-elastic-auth true returns;

Auth is already enabled

The Kibana web interface says;

Kibana server is not ready yet

Grid shows a fault error and the version is still at 2.3.240

salt-call state.highstate shows:

Summary for local

---

Succeeded: 779 (changed=46)
Failed: 2

Total states run: 781
Total run time: 1206.472 s

failed items being:

---

      ID: wait_for_elasticsearch
Function: cmd.run
    Name: so-elasticsearch-wait
  Result: False
 Comment: Command "so-elasticsearch-wait" run
 Started: 21:07:50.551765
Duration: 346381.55 ms
 Changes:
          ----------
          pid:
              22453
          retcode:
              1
          stderr:
          stdout:
              Waiting for value 'green open' at 'https://localhost:9200/_cat/indices/.kibana*' (1/300)
              Server is not ready

and

---

      ID: so-elastalert
Function: docker_container.running
  Result: False
 Comment: One or more requisite failed: elastalert.wait_for_elasticsearch
 Started: 21:13:36.957735
Duration: 0.005 ms
 Changes:

Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://security-onion:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [so_elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [so_elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security"","ApiKey"]}},"status":401}]

so-status shows everything OK, except so-elastalert is missing

Any help to fix would be appreciated, thanks
Stuart

[cm-ops]

Would you be able to share your soup.log?

[The1Waterman]

Please see the attached

soup.log

Thanks