Grid Members Acceptance fails

[izidood]

Version

2.4.30

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32

Storage for /

100GB

Storage for /nsm

500GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hi.

So we did an upgrade on our security onion setup from 2.3 to 2.4.
The manager runs and can be logged into and so on.

But, the Grid only shows the manager, and not all the other nodes we have in our distributed env.
So i went into:
Administration -> Configuration -> Allow Search Nodes to connect to Manager.
put in the IP on one of the search nodes. then after a sync it shows up in pending members, and ready to be accepted.
So i press on (Review) and Accept. after a couple of seconds i get this error and nothing happens.
Timed out waiting for result. This does not necessarily indicate that the command has failed, as a timeout is often caused by a salt high-state running on the manager. Refresh the browser after a couple of minutes to see if the operation succeeded. If not, try the operation again. If the timeout persists, there may be a problem within the cluster. See the Help section of the Security Onion documentation for additional troubleshooting guidance. The Hunt interface may contain additional log messages to further diagnose the issue.

I see it says something about it being salt high-state running on the manager. But i do not think thats the issue, as i dont see it runnning when this problem appears.

Anyone that had any similar problems with this?
OBS. when i go into Hunt i see the sensors, just not in the Grid, and we do get data.

Specs:

• What are the hardware specs of each of those nodes?

Manager Node: VM, 8 cores (no MHz limit), 32GB Memory, 1 x OS Disk 100GB, 1 x NSM Disk 500GB, 1 x NIC.
Search Nodes: VM, 4 Cores (no MHz limit), 64GB Memory, 1 x OS Disk 100GB, 1 x Storage disk 2TB, 1 x NIC.
Receiver Nodes: VM, 4 Cores (no MHz limit), 8GB Memory, 1 x OS Disk 100GB, 1 x NIC.
Forward Nodes: Physical Nodes, 40 Cores, 600GB Memory, 1 x OS Disk 600GB, 1 x NSM Disk 6.6TB, 4 x NIC.

• Does SOC Grid show all services on all nodes as running OK?
  Only the Manager is showing in the Grid with no errors.

• Total amount of nodes: 11
  1 Manager Node
  5 Search Nodes
  2 Receiver Nodes
  3 Forward Nodes

• Do you get any failures when you run sudo salt-call state.highstate?

Yes, 2 errors but related to fleet, which shouldn't be the cause to the nodes not being shown in the Grid.

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[cm-ops]

What returns if you run this command from your manager CLI - sudo salt \* cmd.run "so-status"?

[izidood]

So i figured out how to accept the nodes with the command: salt-key -a
I now see the Accepted members, but not showing in the grid itself.
Then i did a sync and got this: ```
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command: salt-run jobs.lookup_jid 20240105083703854611

So i ran the command: salt-run jobs.lookup_jid and the long number and got the output: 
```[ERROR   ] Unable to connect pusher: Stream is closed.```
so i guess im in the right direction ?, but not sure how to fix that.

[izidood]

i know this thread is getting a bit long, but im still trying to figure it out.
And now i found out that the minion im testing still has the old minion_master.pub key which does not match the masters minion_master.pub key after the the upgrade. i tried to delete the minion_master.pub on the minon and restart the service on the minion, it regenerated the file, but with the same old key. pretty sure this might be the problem ?

[reyesj2]

From the manager and the manager only what is the result of salt '*' test.ping if it returns multiple nodes saying 'true' run salt '*' state.highstate then let us know what if any errors return.

[izidood]

on the manager it only shows the manager itself and no other nodes.

[reyesj2]

This is an upgrade, so your sensors / search nodes also got reimaged to 2.4 correct?

I saw you previously tried accepting salt keys manually, but that isn't going to get the minion setup properly. Best to avoid that.
What is the output of salt-key -L from the manager

[izidood]

This is an upgrade to 2.4 yes, and i guess the other nodes followed the upgrade ? didnt do anything actively on the other nodes under the upgrade. when i run that command i get:
Accepted Key:
All 11 nodes.
But these only shows up here because i did it manually, but doesnt show up in the grid.
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Though i dont see the IDH nodes we are running as well.

[reyesj2]

You reimaged the Manager but not your search nodes / forward nodes? If that is the case they also need to be reimaged and brought up to Security Onion 2.4 that is not an automatic upgrade since it requires a OS reinstall.

[izidood]

yea, we only upgraded the manager, didn't do anything actively on the rest of the nodes, as we thought it would upgrade all the nodes, we'll try to do this on the other nodes tomorrow, I'll make an update in here as solved if this hopefully fixes our problem 🤦‍♂️😊 thanks.

[izidood]

uploading sosetup.log file.
Theres some errors in it when we did the upgrade, maybe someone in here is able to get something informational out of it ?
sosetup.log.txt