Distributed Deployment - Sensor Node not joining

[amma-chadstout]

Version

2.4.60

Installation Method

Network installation on Ubuntu

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4

RAM

32

Storage for /

2tb

Storage for /nsm

2tb

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hello all,

I am rebuilding a distributed deployment for 2.4.60 and am running into an issue. The plan is to have a Manager node, Search node, Fleet node, and 2 Sensor/Forward nodes. I cannot get the second Sensor/Forward node to work properly. When I first attempted to do a network installation onto Ubuntu 22.04, it mostly worked (Strelka failed). So I attempted to redo the install, and now it won't work at all.

On the Sensor node itself, none of the containers are working. Running 'sudo salt-call state.highstate' results in 11 failed. Strelka Filestream, Strelka Manager, Strelka Backend, Strelka Frontend, Strelka Gatekeeper, Strelka Coordinator, so-zeek, so-suricata, so-steno, so-telegraf, so-sensoroni. Each failed with "Failed to pull :5000/security-onion-solutions/so-soc:2.4.60: Error 404: manifest for :5000/security-onion-solutions/so-soc:2.4.60 not found: manifest unknown: manifest unknown" (replacing the various things per entry)

The funny thing is that the troublesome Sensor/Forward node appears in Administration -> Grid Configuration, correctly. However, it does not show up in the "Grid" area. Attempting to run 'sudo so-checkin' or 'sudo salt-call state.highstate', gets me no where. I can ping the Manager node by hostname, and it returns the correct IP. PCAPs show the pings are getting to where they ought to be and are correct.

I think that this whole issue MAY be because of the first failed node, uses the same hostname and IP. Maybe there's a lingering config?

Please let me know if there are any outputs, commands, etc. that I can provide!

Thank you!!!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[amma-chadstout]

Looks like I didn't review similar problems closely enough. This was resolved with running "sudo so-docker-refresh" on the Manager node, then running "sudo so-checkin" on the problematic Sensor node.