What is so-dockerregistry? so-dockerregistry always MISSING

[KhorZL]

Version

2.4.3

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

cloud

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

128

Storage for /

100

Storage for /nsm

12T

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Ask

Any advice on how I can proceed troubleshooting will be of great help. My so-dockerregistry is stuck in a [MISSING] state and /opt/so/log/docker-registry is empty.

Context

I am attempting to install a new Security Onion Distributed set up. This is done completely on AWS.

For some resons, so-dockerregistry is always [MISSING].

I have checked that no process are running on Port 5000 (seems to be an issue in previous discussions).

My errors.log output below:

[ERROR   ] {'image': {'Time_Elapsed': 0.7997114658355713, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.9019663333892822, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.9309172630310059, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.9116146564483643, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.8714284896850586, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
      Result: False
                      - error pulling image configuration: download failed after attempts=6: Forbidden
[ERROR   ] {'image': {'Time_Elapsed': 0.8795206546783447, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.8305606842041016, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.8358209133148193, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.8250625133514404, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] {'image': {'Time_Elapsed': 0.8919458389282227, 'retcode': 1, 'Errors': ['error pulling image configuration: download failed after attempts=6: Forbidden']}}
[ERROR   ] Command '/usr/sbin/so-rule-update' failed with return code: 1
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 67344, 'retcode': 1, 'stdout': '', 'stderr': ''}
      Result: False
                      - error pulling image configuration: download failed after attempts=6: Forbidden
      Result: False

In my so-setup.log I see some issue with docker pull from ghcr.io. I'm also running a squid proxy, but doing a curl returns HTTP 301, so I'm quite sure I'm not blocking any connections :

          ID: so-dockerregistry
    Function: docker_container.running
      Result: False
     Comment: Attempt 1: Returned a result of "False", with the following comment: "Image 'ghcr.io/security-onion-solutions/registry:2.8.2' not present despite a docker pull 
raising no errors"
              Attempt 2: Returned a result of "False", with the following comment: "Image 'ghcr.io/security-onion-solutions/registry:2.8.2' not present despite a docker pull 
raising no errors"
              Attempt 3: Returned a result of "False", with the following comment: "Image 'ghcr.io/security-onion-solutions/registry:2.8.2' not present despite a docker pull raising no errors"
              Attempt 4: Returned a result of "False", with the following comment: "Image 'ghcr.io/security-onion-solutions/registry:2.8.2' not present despite a docker pull raising no errors"
              Image 'ghcr.io/security-onion-solutions/registry:2.8.2' not present despite a docker pull raising no errors
     Started: 09:43:55.518225
    Duration: 149481.36200000002 ms
     Changes:   
              ----------
              image:
                  ----------
                  Errors:
                      - error pulling image configuration: download failed after attempts=6: Forbidden
                  Time_Elapsed:
                      0.8714284896850586
                  retcode:
                      1

I also see some other errors like:

    Function: http.wait_for_successful_query
        Name: http://test-domain:4434/
      Result: True
     Comment: Status 404 was found.
                status: 404
                error: HTTP 404: Not Found
                body: 404 page not found

     Started: 09:54:01.726411
    Duration: 5.77 ms
     Changes:   

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[KhorZL]

Also, running so-docker-refresh, I see the error dial tcp 127.0.0.1:5000: connect: connection refused

This is a manager, so we can proceed.
gpg: key 7C1060B7FE507013: "Security Onion Solutions, LLC <[email protected]>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
Downloading so-curator
Executing command with retry support: docker pull ghcr.io/security-onion-solutions/so-curator:2.4.30
Results: 2.4.30: Pulling from security-onion-solutions/so-curator
Digest: sha256:2e18cfaf2a61fcf50c356c861feb401abb373fe693bdc95a7f8cece7b000f279
Status: Image is up to date for ghcr.io/security-onion-solutions/so-curator:2.4.30
ghcr.io/security-onion-solutions/so-curator:2.4.30 (0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   566  100   566    0     0    895      0 --:--:-- --:--:-- --:--:--   895
The push refers to repository [test-domain:5000/security-onion-solutions/so-curator]
Get "http://test-domain:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused
Downloading so-elastalert
Executing command with retry support: docker pull ghcr.io/security-onion-solutions/so-elastalert:2.4.30
Results: 2.4.30: Pulling from security-onion-solutions/so-elastalert
Digest: sha256:8460a63c754304279eec24aa37c226a6aa8520af753c321c5df0e3318ddf4c7a
Status: Image is up to date for ghcr.io/security-onion-solutions/so-elastalert:2.4.30
ghcr.io/security-onion-solutions/so-elastalert:2.4.30 (0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   566  100   566    0     0    905      0 --:--:-- --:--:-- --:--:--   904
The push refers to repository [test-domain:5000/security-onion-solutions/so-elastalert]
Get "http://test-domain:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

[KhorZL]

Seems like it is indeed a proxy issue.

Removing proxy settings from Docker daemon /etc/systemd/system/docker.service.d/http-proxy.conf fixed it (now it's pulling directly from the internet).

http-proxy.conf now looks like this

[Service]
Environment=

This is a work-around solution for the short-term. But I'm wondering what needs to be whitelisted on the forward proxy side?
I'm running Squid as my proxy, with the following

acl allowed_ports port 80
acl allowed_ports port 443
acl allowed_domain dstdomain ghcr.io

[dougburks]

Have you checked the documentation?
https://docs.securityonion.net/en/2.4/firewall.html#internet-communication

[KhorZL]

Thanks @dougburks ! This was it. I was missing pkg-containers.githubusercontent.com. Cheers!