Fleetdm showing hosts as offline, including the manager

[Pencat89]

I'm installing a new SO on ubuntu server 20.04.6 LTS.
Fleetdm was working to a point, then it stopped working shortly after I joined a search node in.
Looked at the documentation, checked the logs (sudo docker logs so-fleet), and not giving any information. Running SO 2.3.260.
Tried disabling ufw and made no difference.

Any ideas on where to start troubleshooting?

[Pencat89]

Anyone else having issues with this? Rebuild it on ubuntu and same thing happened.

[Pencat89]

Does not register new hosts. Going to test if it's related to 2.3.260 by installing from iso image

[Pencat89]

Tried installing a new deployment on both 2.3.240 and 2.3.260. Had grid/grafana issues along with the fleetdm.

Went back to unbuntu.

Does anyone have instructions on how to reregister fleetdm on the manager? They seem to be in the docker containers.

[Pencat89]

Pretty sure when the security onion script completes the install and requests for a reboot, something in the clean up script is killing fleetdm agent, so when it reboots, it's no longer registering the manager as online.

[Pencat89]

Confirmed it happens to other nodes joining an existing deployment. It's fine up the to point where you reboot, but as soon as you reboot, it no longer comes back as online in fleetdm and just remains offline forever.

Can we get this fixed Devs?

[Pencat89]

I found the problem. For some reason one of my sensors upgraded to 5.9.1 and it started working. All the version ons 4.5.1 are not working.

How do we trigger off an osquery upgrade on those nodes?

[cm-ops]

You could try this #9894

[Pencat89]

Thanks for the details. I originaly had trouble with the compressed file

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Ended up getting the working osqueryd from the one that mysteriously upgraded
scp to the non working nodes and put it in the user directory

As su, ran the following and it worked
Note: Replace user with your user location

cd /usr/local/so-launcher/bin
systemctl stop launcher.so-launcher.service
cp /home/user/osqueryd /usr/local/so-launcher/bin/osqueryd
systemctl restart launcher.so-launcher.service

This bug needs to be fixed in future versions

[Pencat89]

I think it did an update or something and it broke again.