so-mysql ERROR

[KhemaisKebaili]

When I execute so-status , I notice that so-mysql is not started correctly as it shows an error ,also so-playbook and so-fleet since they depend on the so-mysql container.
When I try to read the container's logs I see that the container is stuck in the starting phase :
[root@----------]# docker logs so-mysql
[Entrypoint] MySQL Docker Image 5.7.24-1.1.8
This from command line /etc/mypass
[Entrypoint] Starting MySQL 5.7.24-1.1.8
[Entrypoint] MySQL Docker Image 5.7.24-1.1.8
This from command line /etc/mypass
[Entrypoint] Starting MySQL 5.7.24-1.1.8

How can I try to troubleshoot this? rebooting my Security Onion instance doesn't help at all.

[KhemaisKebaili]

[root@________________]# so-mysql-start --force

Starting mysql...

This could take a while if another Salt job is running.
Run this command with --force to stop all Salt jobs before proceeding.

Force-stopping all Salt jobs before proceeding

local:
- Signal 9 sent to job 20230530074615240644 at pid 745
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] {'stdout': 'until nc -z 192.168.1.11 3306; do sleep 1; done : Timed out after 600 seconds', 'stderr': '', 'retcode': 1, 'pid': 8238}
local:

      ID: mysqlpkgs
Function: pkg.installed
  Result: True
 Comment: All specified packages are already installed
 Started: 08:04:26.714378
Duration: 753.186 ms
 Changes:

---

      ID: mysqletcdir
Function: file.directory
    Name: /opt/so/conf/mysql/etc
  Result: True
 Comment: The directory /opt/so/conf/mysql/etc is in the correct state
 Started: 08:04:27.471335
Duration: 2.625 ms
 Changes:

---

      ID: mysqlpiddir
Function: file.directory
    Name: /opt/so/conf/mysql/pid
  Result: True
 Comment: The directory /opt/so/conf/mysql/pid is in the correct state
 Started: 08:04:27.474104
Duration: 1.438 ms
 Changes:

---

      ID: mysqlcnf
Function: file.managed
    Name: /opt/so/conf/mysql/etc/my.cnf
  Result: True
 Comment: File /opt/so/conf/mysql/etc/my.cnf is in the correct state
 Started: 08:04:27.475676
Duration: 23.279 ms
 Changes:

---

      ID: mysqlpass
Function: file.managed
    Name: /opt/so/conf/mysql/etc/mypass
  Result: True
 Comment: File /opt/so/conf/mysql/etc/mypass is in the correct state
 Started: 08:04:27.499100
Duration: 11.243 ms
 Changes:

---

      ID: mysqllogdir
Function: file.directory
    Name: /opt/so/log/mysql
  Result: True
 Comment: The directory /opt/so/log/mysql is in the correct state
 Started: 08:04:27.510483
Duration: 1.561 ms
 Changes:

---

      ID: mysqldatadir
Function: file.directory
    Name: /nsm/mysql
  Result: True
 Comment: The directory /nsm/mysql is in the correct state
 Started: 08:04:27.512166
Duration: 1.592 ms
 Changes:

---

      ID: so-mysql
Function: docker_container.running
  Result: True
 Comment: Created container 'so-mysql'
 Started: 08:04:27.515206
Duration: 5757.944 ms
 Changes:
          ----------
          container_id:
              ----------
              added:
                  2dfe0b03f673451fea2645fe04d4569624463cc22784088d90af4625fbbda9b5
          state:
              ----------
              new:
                  running
              old:
                  None

---

      ID: so-mysql
Function: cmd.run
    Name: until nc -z 192.168.1.11 3306; do sleep 1; done
  Result: False
 Comment: Command "until nc -z 192.168.1.11 3306; do sleep 1; done" run
 Started: 08:04:33.277896
Duration: 600017.816 ms
 Changes:
          ----------
          pid:
              8238
          retcode:
              1
          stderr:
          stdout:
              until nc -z 192.168.1.11 3306; do sleep 1; done : Timed out after 600 seconds

---

      ID: so-mysql
Function: module.run
  Result: False
 Comment: One or more requisite failed: mysql.so-mysql
 Started: 08:14:33.302890
Duration: 0.012 ms
 Changes:

---

      ID: append_so-mysql_so-status.conf
Function: file.append
    Name: /opt/so/conf/so-status/so-status.conf
  Result: True
 Comment: File /opt/so/conf/so-status/so-status.conf is in correct state
 Started: 08:14:33.303365
Duration: 17.985 ms
 Changes:

Summary for local

Succeeded: 9 (changed=2)
Failed: 2

Total states run: 11
Total run time: 606.589 s

[cm-ops]

Have you changed the IP address? If your mysql container is running are you able to login with the following command? docker exec -it so-mysql mysql -u root -p (use the password from salt-call pillar.get secrets for mysql)

[KhemaisKebaili]

I haven't changed anything , I had to reboot my standalone instance and upon reboot mysql was giving me trouble.
When consulting /opt/so/log/mysql/mysqld.log I found this log:
2023-05-30T00:02:14.398766Z 0 [ERROR] InnoDB: Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint 2631580999 and the end 2631582417.
2023-05-30T00:02:14.398774Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-05-30T00:02:15.000224Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-05-30T00:02:15.000338Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-05-30T00:02:15.000356Z 0 [ERROR] Failed to initialize builtin plugins.
2023-05-30T00:02:15.000367Z 0 [ERROR] Aborting

So I did the following :
docker stop so-mysql
docker rm so-mysql
rm -rf /nsm/mysql

After that , the mysql container was running with no issues, however both fleet and playbook were unable to start..
I did a so-playbook-reset, starting fresh.. so playbook would get started even if it meant I lose all my detection logic :(.
Playbook works just fine for now , however fleet is still giving me trouble

executing so-fleet-setup gives me this message: Username or Password was not set - exiting now.
I'd like to mention that under /nsm there's a a mysql_old_bunchofnumbersmaybeatimetamp folder containing two folders , one for playbook and the other for fleet, if it contains the configuration for both playbook and fleet as they were is there a way to revert? copying the folders doesn't seem to do it. If not , how can I setup fleet correctly

Thanks in advance for your help

[cm-ops]

Try so-fleet-setup "myemail" "mypassword", your password needs to be 12 chars + 1 number + 1 special. Restart SOC afterwards.

[KhemaisKebaili]

It did resolve my issue , thanks a lot!