Upgrade issue: Security Onion stopped workig "Appliance images unavailable"

[urbanovits]

Since the latest upgrade to 2.3.50 Sec Onion stopped working.
Grid message:
Appliance images unavailable.

Nothing has displayed, No alerts, nothing at all.

SSH-into server
and
sudo so-status
result:

Checking Docker status

Docker ----------------------------------------------------------------------------------------- [ OK ]    

Checking container statuses

so-aptcacherng --------------------------------------------------------------------------------- [ OK ]    
so-cortex -------------------------------------------------------------------------------------- [ OK ]    
so-curator ------------------------------------------------------------------------------------- [ OK ]    
so-dockerregistry ------------------------------------------------------------------------------ [ OK ]    
so-elastalert ---------------------------------------------------------------------------------- [ OK ]    
so-elasticsearch ------------------------------------------------------------------------------- [ OK ]    
so-filebeat ------------------------------------------------------------------------------------ [ OK ]    
so-fleet --------------------------------------------------------------------------------------- [ OK ]    
so-grafana ------------------------------------------------------------------------------------- [ OK ]    
so-idstools ------------------------------------------------------------------------------------ [ OK ]    
so-influxdb ------------------------------------------------------------------------------------ [ OK ]    
so-kibana -------------------------------------------------------------------------------------- [ OK ]    
so-kratos -------------------------------------------------------------------------------------- [ OK ]    
so-logstash ------------------------------------------------------------------------------------ [ OK ]    
so-mysql --------------------------------------------------------------------------------------- [ OK ]    
so-nginx --------------------------------------------------------------------------------------- [ OK ]    
so-playbook ------------------------------------------------------------------------------------ [ OK ]    
so-redis --------------------------------------------------------------------------------------- [ OK ]    
so-sensoroni ----------------------------------------------------------------------------------- [ OK ]    
so-soc ----------------------------------------------------------------------------------------- [ OK ]    
so-soctopus ------------------------------------------------------------------------------------ [ OK ]    
so-steno --------------------------------------------------------------------------------------- [ OK ]    
so-strelka-backend ----------------------------------------------------------------------------- [ OK ]    
so-strelka-coordinator ------------------------------------------------------------------------- [ OK ]    
so-strelka-filestream -------------------------------------------------------------------------- [ OK ]    
so-strelka-frontend ---------------------------------------------------------------------------- [ OK ]    
so-strelka-gatekeeper -------------------------------------------------------------------------- [ OK ]    
so-strelka-manager ----------------------------------------------------------------------------- [ OK ]    
so-suricata ------------------------------------------------------------------------------------ [ OK ]    
so-telegraf ------------------------------------------------------------------------------------ [ OK ]    
so-thehive ------------------------------------------------------------------------------------- [ OK ]    
so-thehive-es ---------------------------------------------------------------------------------- [ OK ]    
so-wazuh --------------------------------------------------------------------------------------- [ OK ]    
so-zeek ---------------------------------------------------------------------------------------- [ OK ]    

then
sudo salt * so.status

No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received

/opt/so/log/salt/minion

minion log:
2023-11-28 09:11:44,342 [salt.state :2244][INFO ][4175] Completed state [query_updatwebhooks] at time 09:11:44.342289 (duration_in_ms=13.785)
2023-11-28 09:11:44,342 [salt.state :2057][INFO ][4175] Running state [query_updatepluginurls] at time 09:11:44.342667
2023-11-28 09:11:44,343 [salt.state :2089][INFO ][4175] Executing state mysql_query.run for [query_updatepluginurls]
2023-11-28 09:11:44,375 [salt.state :321 ][ERROR ][4175] An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 438, in _connect
dbc = MySQLdb.connect(**connargs)
File "/usr/lib64/python3.6/site-packages/MySQLdb/init.py", line 85, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in init
self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/state.py", line 2172, in call
*cdata["args"], **cdata["kwargs"]
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1235, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2268, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2283, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2316, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/states/mysql_query.py", line 376, in run
query_result = salt["mysql.query"](database, query, **connection_args)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1235, in call
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2268, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/init.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 2283, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 773, in query
dbc = _connect(**connection_args)
File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 444, in _connect
except MySQLdb.err.InternalError as exc:
AttributeError: module 'MySQLdb' has no attribute 'err'

/opt/so/log/salt/so-salt-minion-check log
no suspicious event

I cannot overcome, please help.

[dougburks]

Grid message:
Appliance images unavailable.

This most likely just means that you're running Security Onion on hardware other than official Security Onion Solutions appliances. From https://docs.securityonion.net/en/2.3/grid.html:

Since the latest upgrade to 2.3.50 Sec Onion stopped working.

Are you saying you just upgraded to 2.3.50? That's a fairly old version, released in 2021:
https://blog.securityonion.net/2021/04/security-onion-2350-now-available.html

The latest version of 2.3.x is 2.3.280 (released yesterday):
https://blog.securityonion.net/2023/11/security-onion-23280-now-available.html

So you might try updating to 2.3.280 or performing a fresh installation of 2.3.280. However, please note that Security Onion 2.3 reaches End Of Life on April 6, 2024:
https://blog.securityonion.net/2023/10/6-month-eol-notice-for-security-onion-23.html

So you might want to take this opportunity to migrate to 2.4.30:
https://blog.securityonion.net/2023/11/security-onion-2430-now-available.html