Grid Members Not Populating

[DrOnionMaster]

Hey all,

For context, I have a freshly running instance of a distributed grid consisting of a manager, search, receiver, and forward node. They are all up and running smoothly, everything is connected and there are no errors in any of the logs I checked. I am having a problem with the grid members not showing up under the "Grid" tab in SOC. All of my nodes are showing up in the "Accepted Members" in Administration > Grid Members, but they are not populated in the Grid tab.

Also... I confirmed that there is nothing being blocked on our firewall and all the ports that are supposed to be talking are open to communicate between the nodes, so I know this isn't a firewall issue. I saw someone with a similar issue talking about sensoroni being the issue but can't seem to figure it out.

All help is much appreciated, thanks!

[reyesj2]

From the manager cli can you run

sudo salt '*' so.status

All your accepted grid members should return with good status for so-status. If they aren't there is probably something stopping the minions from completing their initial highstate.

How did you add these minions to your grid? Was it from the GUI under grid members ?

[DrOnionMaster]

They were added during the initial installation, when everything was finally stood up, they were already accepted in the GUI. I did however have to manually accept the keys by running sudo salt-key --list all and sudo salt-key -a saltmaster.

[dougburks]

They shouldn't have been already accepted in the GUI and if they were then you shouldn't have had to run those salt commands.

Have you ever re-run setup on any of these machines?

You could try running sudo so-checkin on the manager and then on each of the other nodes.

If all else fails, you can always wipe and reinstall using the new 2.4.70 ISO image:
https://blog.securityonion.net/2024/05/security-onion-2470-now-available.html

[DrOnionMaster]

I haven't rerun the setup... is there a command for that? Or is it a matter of nuking them and restarting from a new ISO?

Also, curious what running soup would do?

I have elastic agents deployed already, will reinstalling with a new ISO break these?

Also, I tried running sudo so-checkin on all of the nodes, but it can only be run on the manager. When I try running that command on the other nodes it says, "sudo: so-checkin: command not found".

[DrOnionMaster]

Update I was able to successfully update to 2.4.7, this has not resolved any of the issues.

[dougburks]

If so-checkin won't run on the nodes, then they never fully connected to the manager. I would try wiping a node and re-installing using the 2.4.70 ISO image.

[linuxlurak]

Same here, upgraded SO from 2.4.60 to 2.4.70, Grid Members added before, now showing as accepted Members but did not show up in grid tab. I tried sudo salt-run mine.get \* x509.get_pem_entries && sudo salt -C "G@role:so-idh" state.highstate according to #12979 and results were ok. But grid members still not showing up. After issuing a so-sensoroni-restart on the idh-nodes they show up but as faulty...

Any hints for how to investigate further? Which logs to consult?

[dougburks]

@linuxlurak From #1720:

Start a new discussion instead of replying to somebody else's discussion
Please search to see if you can find similar discussions that may help you. However, in order to avoid confusion, please do NOT reply to somebody else's discussion with your own issue. Instead, please start a new discussion and in that new discussion you can provide a hyperlink to the related discussion.

[DrOnionMaster]

I was able to resolve this issue by deleting the nodes from the "Accepting members" section of the SOC and rebooting my manager node. The nodes then came up as unaccepted and I simple re-accepted them and everything works.