Failed upgrade from 2.4.50 to 2.4.60, Soup logs point to problems with upgrading Certs and Docker Registry is now broken

[jollysig]

Version

2.4.60

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

128gb

Storage for /

100gb

Storage for /nsm

1tb

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Ran soup on a distributed installation of 2.4.50. Soup failed 27 times. "Salt master must permit peers to call the sign_remote_certificate function" or "One of more requisite failed" was the most common error types. Both of these error types seem to be dealing with certificates.

Tried rebooting and running Soup again hoping that it would fix the errors from the first try. That didn't work and now I'm getting "Error response form daemon: No such container: so-dockerregistry". This error shows when I run soup, directly underneath "Updating dockers to 2.4.60" but doesn't seem to be showing in the second soup.log.

Running So-status on the Manager returns "System appears to be starting. No highstate has completed since the system was restarted.". Running a "salt-call state.highstate" returns no errors. The webserver is not running so I can't look at Grid.

I have a Manager, 2 search nodes, and 2 sensors. Running so-status on the search nodes and sensors shows everything is running on the other servers so I'm hoping this has only effected the Manager.

I'm attaching the Soup logs from the first try of Soup (soup.log1) and the second (soup.log2). Any help getting this back online would be greatly appreciated.

soup.log1.txt
soup.log2.txt

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[jollysig]

Thanks to #12078 and @Bal33p for posting the solution to his problem with the docker registry. I was able to use that to get my Security Onion cluster fixed and upgraded from 2.4.5 to 2.4.6:

So my thoughts were there has to be something wrong with so-dockerregistry since I keep seeing that in the soup logs. So I start looking in the forums for anyone else having that problem with 2.4.x or later. Found @Bal33p's "sodockerregistry will not start" #12078. I figured I would use his solution to try to get my so-dockerregistry up and running:

Logged into my manager and elevated to Root

sudo su

listed the Docker containers

docker ps

That showed that so-dockerregistry wasn't a listed container so let's try to fix that:

salt-call state.apply registry

another "docker ps" and now the registry is showing as a container. Awesome, small win.

so-docker-refresh

Refresh completes, no errors.

Webserver on the Manager starts working correctly and I can log into the SOC console and Kibana again.

so-status

Shows everything is running no errors. Grid is showing no faults. Now to see if we can finish the upgrade without issue

soup

Soup runs with no errors and everything is working correctly and showing as being on 2.4.60

Awesome!

[RagnarSTS]

I also had errors; missing logstash, not processing new data.
I performed the same tasks you did and they were also fixed. I have a great hatred for docker. I think this is the first time I've fixed an upgrade error without having to reload.

[jollysig]

@RagnarSTS I'm glad that fixed your problem. I've been running Security Onion since 2015, and truthfully, this was the first time I've ever had a problem with an in-place upgrade that running "soup" again didn't fix. I've probably installed Security Onion over 300 hundred times across 100 different locations and this was my first-time having issues with Docker.

@dougburks @defensivedepth I will say that finding this solution took me a while to find searching through the forum. Maybe a version of troubleshooting steps can get added to the "help" section of the documentation?

[dougburks]

@jollysig I've updated the soup page of the documentation to include a link to this discussion:
https://docs.securityonion.net/en/2.4/soup.html#docker-registry

In the future, please keep in mind this guideline from #1720:

Please do not tag an individual in a discussion unless that individual has already volunteered to help you in that discussion.

Thanks.