Docker containers failing to update during upgrade from 2.3.91 to 2.3.130

[radman13666]

I recently did an upgrade from 2.3.91 to 2.3.130, but I got an error on multiple containers that was as below:

ID: so-sensoroni
Function: docker_container.running
Result: False
Comment: Failed to pull so-manager-node:5000/security-onion-solutions/so-soc:2.3.130: Error 404: manifest for so-manager-node:5000/security-onion-solutions/so-soc:2.3.130 not found: manifest unknown: manifest unknown
Started: 10:13:05.440424
Duration: 235.316 ms
Changes:

On rebooting the manager node, the containers failed to come up:

# so-status

Checking Docker status

    Docker -------------------------------------------------------------------------------------------------------------------- [ OK ]

Checking container statuses

    so-aptcacherng ------------------------------------------------------------------------------------------------------------ [ OK ]
    so-cortex ----------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-dockerregistry --------------------------------------------------------------------------------------------------------- [ OK ]
    so-elastalert ------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-elasticsearch ---------------------------------------------------------------------------------------------------- [ STARTING ]
    so-filebeat --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-fleet ------------------------------------------------------------------------------------------------------------ [ STARTING ]
    so-grafana ---------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-idstools --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-influxdb --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-kibana ----------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-kratos ----------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-logstash --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-mysql ------------------------------------------------------------------------------------------------------------ [ STARTING ]
    so-nginx ------------------------------------------------------------------------------------------------------------ [ STARTING ]
    so-playbook --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-redis ------------------------------------------------------------------------------------------------------------ [ STARTING ]
    so-sensoroni -------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-soc -------------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-soctopus --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-telegraf --------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-thehive ---------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-thehive-es ------------------------------------------------------------------------------------------------------- [ STARTING ]
    so-wazuh ------------------------------------------------------------------------------------------------------------ [ STARTING ]

Checking the docker containers, all were not using the 2.3.130 release but remained on the 2.3.91 release.

# docker container list -a
CONTAINER ID   IMAGE                                                               COMMAND                  CREATED         STATUS                          PORTS                                                                                NAMES
06f62bc110e2   ghcr.io/security-onion-solutions/registry:latest                    "/entrypoint.sh /etc…"   28 hours ago    Up 26 hours                     0.0.0.0:5000->5000/tcp                                                               so-dockerregistry
dc74c146e5ff   so-manager-node:5000/security-onion-solutions/so-soc:2.3.91              "/opt/sensoroni/sens…"   8 days ago      Exited (0) 26 hours ago                                                                                              so-soc
329a4ef84008   so-manager-node:5000/security-onion-solutions/so-fleet:2.3.91            "/startfleet.sh"         2 weeks ago     Exited (255) 26 hours ago       0.0.0.0:8080->8080/tcp                                                               so-fleet
5f40ebda8491   so-manager-node:5000/security-onion-solutions/so-mysql:2.3.91            "/entrypoint.sh mysq…"   3 weeks ago     Exited (255) 26 hours ago       0.0.0.0:3306->3306/tcp, 33060/tcp                                                    so-mysql
b65b11007baf   so-manager-node:5000/security-onion-solutions/so-redis:2.3.91            "redis-server /usr/l…"   3 weeks ago     Exited (0) 26 hours ago                                                                                              so-redis
eff990893988   so-manager-node:5000/security-onion-solutions/so-thehive:2.3.91          "/opt/thehive/bin/so…"   3 weeks ago     Exited (255) 26 hours ago       0.0.0.0:9000->9000/tcp                                                               so-thehive
439cf34a1f58   so-manager-node:5000/security-onion-solutions/so-wazuh:2.3.91            "/entrypoint.sh"         13 months ago   Exited (255) 26 hours ago       0.0.0.0:1514-1515->1514-1515/tcp, 0.0.0.0:55000->55000/tcp, 0.0.0.0:1514->1514/udp   so-wazuh
db31f7161466   so-manager-node:5000/security-onion-solutions/so-grafana:2.3.91          "/run.sh"                16 months ago   Exited (0) 26 hours ago                                                                                              so-grafana
5c70829f7efb   so-manager-node:5000/security-onion-solutions/so-nginx:2.3.91            "/docker-entrypoint.…"   16 months ago   Exited (0) 26 hours ago                                                                                              so-nginx
8bba77fd2eaa   so-manager-node:5000/security-onion-solutions/so-telegraf:2.3.91         "/entrypoint.sh tele…"   16 months ago   Exited (0) 26 hours ago                                                                                              so-telegraf
00aa7760c6d6   so-manager-node:5000/security-onion-solutions/so-soc:2.3.91              "/opt/sensoroni/sens…"   16 months ago   Exited (0) 26 hours ago                                                                                              so-sensoroni
5c0a55916f4a   so-manager-node:5000/security-onion-solutions/so-thehive-cortex:2.3.91   "/opt/cortex/bin/cor…"   16 months ago   Exited (255) 26 hours ago       0.0.0.0:9001->9001/tcp                                                               so-cortex
68d2f4ec16f4   so-manager-node:5000/security-onion-solutions/so-thehive-es:2.3.91       "/bin/tini -- /usr/l…"   16 months ago   Exited (143) 26 hours ago                                                                                            so-thehive-es
ea37220c778d   so-manager-node:5000/security-onion-solutions/so-filebeat:2.3.91         "/usr/local/bin/dock…"   16 months ago   Exited (0) 26 hours ago                                                                                              so-filebeat
ada202e6b79e   so-manager-node:5000/security-onion-solutions/so-influxdb:2.3.91         "/entrypoint.sh infl…"   16 months ago   Exited (0) 26 hours ago                                                                                              so-influxdb
d0c538e556c3   so-manager-node:5000/security-onion-solutions/so-soctopus:2.3.91         "gunicorn -b 0.0.0.0…"   16 months ago   Exited (255) 26 hours ago       0.0.0.0:7000->7000/tcp                                                               so-soctopus
bbc3c60de314   so-manager-node:5000/security-onion-solutions/so-elastalert:2.3.91       "/opt/elastalert/run…"   16 months ago   Exited (255) 26 hours ago                                                                                            so-elastalert
0e28518fb21b   so-manager-node:5000/security-onion-solutions/so-playbook:2.3.91         "/docker-entrypoint.…"   16 months ago   Exited (255) 26 hours ago       0.0.0.0:3200->3000/tcp                                                               so-playbook
ccc3311783f5   so-manager-node:5000/security-onion-solutions/so-idstools:2.3.91         "./entrypoint.sh"        16 months ago   Exited (255) 26 hours ago                                                                                            so-idstools
a138665f5e89   so-manager-node:5000/security-onion-solutions/so-acng:2.3.91             "/sbin/so-entrypoint…"   16 months ago   Up 26 hours                     0.0.0.0:3142->3142/tcp                                                               so-aptcacherng
1d347070c5ae   so-manager-node:5000/security-onion-solutions/so-kratos:2.3.91           "/start-kratos.sh"       16 months ago   Restarting (1) 24 seconds ago                                                                                        so-kratos

I tried to do a re-upgrade, but the Elasticsearch container was not running, so I'm stuck.

Is there are way to update all containers to the 2.3.130 versions easily, or if not, is there way to rollback the upgrade easily?

[cm-ops]

Do you have the 2.3.130 docker images? If you can share your soup.log that might shed some light on issues.