Cluster Servers not communicating?

[cannonesshannah]

I remember this being fussy the last time I set up a CSF cluster, but unfortunately, I didn't think to document how I fixed it and I'm running into the cluster members not communicating.

Context:
I have a group of servers (well, proxmox vms) set up as follow:

38.143.59.195 | azalin.toud.pw
38.143.59.196 | ezra.toud.pw
38.143.59.197 | nwsync.toud.pw
38.143.59.198 | www.toud.pw
38.143.59.199 | wiki.toud.pw
38.143.59.200 | git.toud.pw
38.143.59.201 | chat.toud.pw
38.143.59.202 | forums.toud.pw

The forums get the most web traffic, particularly for bots, so I set it as the master.

My configuration on all nodes is as follows:

CLUSTER_SENDTO = "38.143.59.194,38.143.59.195,38.143.59.196,38.143.59.197,38.143.59.198,38.143.59.199,38.143.59.200,38.143.59.201"
CLUSTER_RECVFROM = "38.143.59.194,38.143.59.195,38.143.59.196,38.143.59.197,38.143.59.198,38.143.59.199,38.143.59.200,38.143.59.201"
CLUSTER_MASTER = "38.143.59.202"
CLUSTER_NAT = ""
CLUSTER_LOCALADDR = ""
CLUSTER_PORT = "7777"

(this is from the 202 master, it obviously isn't sending to itself)

I ensured the cluster_key is identical though I have not included it for security reasons

I can ping the servers normally, so the routes are reachable:

root@nwsync:~# ping 38.143.59.202
PING 38.143.59.202 (38.143.59.202) 56(84) bytes of data.
64 bytes from 38.143.59.202: icmp_seq=1 ttl=64 time=0.391 ms
64 bytes from 38.143.59.202: icmp_seq=2 ttl=64 time=0.268 ms
64 bytes from 38.143.59.202: icmp_seq=3 ttl=64 time=0.477 ms

However cluster ping fails:

Cluster error connecting to 38.143.59.202: Connection refused
Failed to connect to 38.143.59.202

Thinking perhaps the port was not being opened properly by CSF, I added 7777 to the list of ports on ipv4 and 6, in and out, but that didn't seem to change anything.

I tried setting NAT and LOCALADDR to the local address but that didn't affect anything either.

Any idea what might be up?

• May

[Aetherinox]

If you're referring to CLUSTER_PORT, you should not have to add it to TCP_IN / Out. It is automatically added to the list of in/out rules.

The easiest way to determine if the ports are open is to run

sudo iptables -L -n -v

As CSF converts everything you've set up over the iptable rules.

And also, seems obvious, but ensure you restart CSF and then LFD when modifying any of the CLUSTER settings, as the rules will also be rehashed at that time. Sounds like the typical "Turn it off and back on", but I've had it happen from experience due to not paying attention.

Check the ports in iptables first just to rule that out. As for cluster ping rules, are you getting any additional information other than the connection refused? Pings are reported to lfd.log and usually display a lot more information.

[cannonesshannah]

Yes, I mention it only as I did try that - as you say, I shouldn't have to open those port as csf should add them automatically. However when it didn't work properly, my assumption was maybe, for whatever reason, maybe it didn't. Directly adding them did not see a change, however.

I do not find a lfd.log in /var/log/csf/lfd.log or /var/log/lfd.log - should it be elsewhere?

[edit] A thought: should CSF start LFD automatically? CSF seems to be starting itself, but not LFD.

It is not in testing mode.

I seem to remember it should if it's not in testing mode.

[Update]: Manually starting LFD results in it working (makes sense, it has the cluster server IIRC), but it is NOT being brought up/down with CSF despite testing mode being off.

[Aetherinox]

I went through a giant Easter egg hunt of the code.

LFD / CSF starting up greatly depends on the control panel.

Cpanel takes care of this internally, generic installation starts them, but any number of things can cause LFD to not start. Which I think needs to be re-worked. But they're never attempted during the initial install, only enable is attempted.

I'm going to do some clean installs today and sort of give LFD and CSF a series of tests to see exactly what steps are required (aside from TESTING) and what can block LFD from starting up.

For me personally, I always have to manually start LFD.

[cannonesshannah]

This is on a "generic" no CP install if it helps. Since I know I have to manually start it, that's more or less the "fix" - but it would be nice if it consistently bootstrapped itself. For one, it stops us having obscure problem easter egg hunts :)

If anything I can do might help - just ask!

[Aetherinox]

Yeah, I've had my own personal gripes about LFD starting over the years. I can't count how many times I've tried to access the web interface, wondering why the hell I'm timing out. Come to find out, LFD wasn't even running. It's one of the things on my list that is at the higher-end of the priority list.

I just got the Discord server set up. So that's one more thing off the list. And I just re-packed the repo. It was up to 2.70GB because of the blocklist commit history. We're back down to 60MB. So we're slowly working through the list.

I'll go through the generic installer today and do a few installs and see what can be done to mitigate the LFD issue.

[cannonesshannah]

One thought: maybe print a warning if the lfd cluster daemon doesn't seem to be present if you try csf -cp, so it at least makes the likely culprit of malfunctions obvious.

[Aetherinox]

One thought: maybe print a warning if the lfd cluster daemon doesn't seem to be present if you try csf -cp, so it at least makes the likely culprit of malfunctions obvious.

That I can do. Technically, it should be checking LFD's status for any lfd related tasks before it even begins to work. No idea why it's not.