Cannot complete addition of Search node. Search node pillar is missing?

[akibako]

Version

Other (please provide detail below)

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

4 (manager node), 4 (search node)

RAM

24GB (manager node), 16GB (search node)

Storage for /

200GB (manager node), 1TB (search node)

Storage for /nsm

Single partitions. See "Storage for /" above.

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

All nodes are version 2.4.201
In my proof of concept environment, I am unable to complete the addition of a Search node to my Manager (be it a "manager" or "managersearch"). I am deploying this environment with only Manager, Search, and Fleet nodes (no Sensors). I am unsure of the root cause, but in my very limited understanding, it seems to be due to the fact that the Manager never creates the necessary pillar for the Search node, so it is unable to pull down and apply the necessary configuration. I've reproduced this over more than 10 redeployment attempts.

1. Am I correct?
2. Is this a known issue? (although I did not see it listed)
3. Is there a workaround?

The setup on the Search node itself seems to complete successfully.
When I Approve the pending Search node, the SOC UI progress spinner spins for about 15 seconds, but then displays the error "Unable to manage minion; ensure that salt is running on the manager node and check salt logs". Even after waiting 1 hour, and then rebooting the Search node and waiting another 24 hours, the Search node never appears in the Grid. (Incidentally, the same error message appeared when I added a Fleet node, but the Fleet node did eventually appear in the Grid and eventually seemed 100% OK.)

The Manager's status appears fine.

[admin@so-manager ~]$ sudo so-status

                       Security Onion Status
                         Container │ Status  │              Details
───────────────────────────────────┼─────────┼──────────────────────
                 so-dockerregistry │ running │           Up 4 hours
                     so-elastalert │ running │           Up 4 hours
                  so-elastic-fleet │ running │           Up 4 hours
 so-elastic-fleet-package-registry │ running │ Up 4 hours (healthy)
                  so-elasticsearch │ running │           Up 4 hours
                       so-influxdb │ running │ Up 4 hours (healthy)
                         so-kibana │ running │           Up 4 hours
                         so-kratos │ running │           Up 4 hours
                       so-logstash │ running │           Up 4 hours
                          so-nginx │ running │ Up 4 hours (healthy)
                          so-redis │ running │           Up 4 hours
                      so-sensoroni │ running │           Up 4 hours
                            so-soc │ running │           Up 4 hours
                       so-telegraf │ running │           Up 4 hours

 ✔ This onion is ready to make your adversaries cry!

The Search node is able to connect to the Manager at the network level.

[admin@so-search01 ~]$ nc -vz 172.16.7.26 4505
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 172.16.7.26:4505.
Ncat: 0 bytes sent, 0 bytes received in 0.04 seconds.
[admin@so-search01 ~]$
[admin@so-search01 ~]$ nc -vz 172.16.7.26 4506
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 172.16.7.26:4506.
Ncat: 0 bytes sent, 0 bytes received in 0.04 seconds.

The Search node's key has been accepted.

[admin@so-manager ~]$ sudo salt-key -L
Accepted Keys:
so-manager_manager
so-search01_searchnode
Denied Keys:
Unaccepted Keys:
Rejected Keys:

The Manager can ping the Search node.

[admin@so-manager ~]$ sudo salt 'so-search01_searchnode' test.ping
so-search01_searchnode:
    True

But while on the Manager, attempting to bring up a highstate on the Search node fails.

[admin@so-manager ~]$ sudo salt 'so-search01_searchnode' state.highstate
so-search01_searchnode:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'minions.so-search01_searchnode' in environment 'base' is not available on the salt master
----------
    Specified SLS 'minions.adv_so-search01_searchnode' in environment 'base' is not available on the salt master
ERROR: Minions returned with non-zero exit code

The Manager seems to be missing the pillar for the Search node.

[admin@so-manager ~]$ sudo salt-run config.get pillar_roots
base:
    - /opt/so/saltstack/local/pillar
    - /opt/so/saltstack/default/pillar
[INFO    ] Runner completed: 20260222125320482085
[admin@so-manager ~]$ sudo ls -lah /opt/so/saltstack/local/pillar/minions
total 8.0K
drwxr-x---.  2 socore socore   70 Feb 22 02:58 .
drwxr-x---. 38 socore socore 4.0K Feb 22 02:58 ..
-rw-r-----.  1 socore socore    0 Feb 22 02:58 adv_so-manager_manager.sls
-rw-r-----.  1 socore socore 1.1K Feb 22 03:55 so-manager_manager.sls

While likely not recommended, I was tentatively able to work around this by creating my own pillar (although I was not 100% sure about the proper content) at:
/opt/so/saltstack/local/pillar/minions/so-search01_searchnode.sls
/opt/so/saltstack/local/pillar/minions/adv_so-search01_searchnode.sls

This allowed the Search node to at least pull down and apply enough configuration to get Sensoroni running and the Search node then appeared in the Grid. However, nginx never came online (likely certificate signing problems), and I also ran into a problems on the Search node where "sudo salt-call pkg.list_holds" failed because the 'python3-dnf-plugin-versionlock' package was not installed. This was my first major sign that I was moving away from simply "bug" territory, and more toward "yeah, something is either truly broken, or I'm doing it wrong", so I decided to stop and check in here.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Looks like there was an issue between accepting the salt-key, which was successful, and the minion pillar file creation. If you see something like this check the /root/sosetup.log for any issues. You can also run sudo so-minion -o=add -m=so-search01_searchnode on your manager node, that would create the minion pillar file correctly and allow the search node to highstate.

[akibako]

@cm-ops Thank you for pointing me in the right direction! :D
More testing revealed that there is a bug when node descriptions contain both spaces and numbers. I've filed #15540 .

Workarounds are to do any of the following:
a. do not set a node description
b. use underscores or hyphens instead of spaces in a node description
c. do not use numbers after spaces in a node description (however note that only the characters before the first space will actually be set as the node's description in the UI)

In my case, I was using the following node descriptions. The Search node would fail, whereas the Fleet node would (mostly) succeed because it did not contain a number after a space.

• Search node desccription: "Security Onion search node 1" --> failed
• Fleet node description: "Security Onion fleet node" --> succeeded, but the description in the UI became merely 'Security'

The root cause occurs within the getinstallinfo() function of /sbin/so-minion:

1. xargs will parse the node description using the spaces as delimiters
2. when a number follows a space, export will then throw an error because a number is not a valid variable name