The main node should keep a percentage of connections to nodes moving with the chain

[brause]

In the recent dusting events i read something that seems to be a good idea for improving the main node. It would have kept my main node online. It has a lots of cpu, memory and the db on ssd to keep easily afloat. Instead it got "knocked off". Well sort of. My node was full of fallen behind connections so i fell behind too.

I propose:
Monitor the list of active nodes if they move with the chain. Then if the amount of nodes that is not moving with the chain is above 70%(that is debatable off course) ban some 10% of the slow ones for a while to make room for new connections.
Of course we need to help people syncing up but it should be balanced.

Thanks for the good work, by the way. Its an amazing product and a fine piece of engineering working like this out of the box.

[Jacek-ghub]

I think that you are misreading the state of your node. It is not really that your node has plenty of resources, but rather the chia code is bad, and cannot adjust to network conditions. If you think about it, when you start syncing from scratch (removing your db), it will take few days to get you synchronized. However, when you calculate your data rates, it will be below 2Mbps download. That would imply that you really need just one fully synced node to keep up with the flow, and not put too much strain on that one node.

A big part of the problem is that chia is using some garbage numbers to control the network handling (e.g., 80 peers). No one who is sane puts such numbers there, as from one point of view, there are different nodes there (more or less capable), and the network conditions change (more or less transactions per second). Any networking code needs to dynamically adjust to what is at hand.

Also, another way of looking at that is that your peers should reflect the state of the network (i.e., when everything is good, it should be ~100% of synced nodes, and during dust storms about 2x-4x * network_space_drop "lagging" nodes. If a node has more of those "lagging" nodes, it implies that it just cannot handle the traffic (the number of connected nodes).

Also, when you start seeing nodes that don't have any heights (either none, or 0), that implies that your node is making connections, but is not able to fully handshake with those peers - basically, your node is SOL, yet still it is scheduling those peers for more work - putting strain on db side.

The best way for your node is to just drop the number of peers in your config.yaml (say by half, if not, then half again, etc.). It is better to run a node with limited number of peers, rather than to have a dead node on the network.

Yet another way of looking at that 80 peers number is from p2p topology. It is enough to have 3 peers to start building a network. Although, that will be a not really resilient network. On the other hand, if every node connects to every node, that would be a super resilient network. With those two extremes, chia developer picked that 80 from a hat, as a sort of compromise, without any understanding of what that implies.

[karstenbruch]

@Jacek-ghub i do not see how that is relevant to my proposal. Actually i had two nodes being stale after the last dust storm. One with 30 connections and one with the standard 80(or was it 60?).

If none of your peers do move anymore you are coming to a halt yourself.

[Jacek-ghub]

During December dust storms the net space dropped about 10% each time. Assuming that most of the farms affected were small ones, let's say that about 50% of nodes were hosed, and couldn't properly exchange data (let's call them "stalled" nodes). That implies that 50% of nodes were not affected at all.

If for whatever reason connection drops, either the node that lost such connection, or some other node asks the introducer host for a list of peers to connect to. We should assume that such list should have 50% of good and 50% of stalled nodes. Therefore, the replacement nodes should be coming not as 100% stalled ones. The fact that both of your nodes ended up with 100% staled nodes would imply that either: 1. your nodes were actively looking only for those stalled nodes (what implies some magic on the side of your node), or 2. that your node just couldn't handle the traffic, and actually couldn't handle the data exchanges with all connected nodes (what is a simpler interpretation).

You are trying to use #1 explanation, and that is just holier-than-thou approach, that assumes that your node is really superior, and nothing affects it, where everything else is just not-performing junk.

Also, the fact that you are trying to invent another magic number (70%, or whatever it is), implies that you really do not understand how many good nodes you need, and are just grasping for some solution, but unfortunately, that is a wrong direction.

A much cleaner / simpler approach is to have the network protocol properly implement some sort of backing off mechanisms that instead would be automatically scaling down the number of connections, thus preventing most, if not all nodes from falling behind.

Therefore, "If none of your peers do move anymore" - that implies that your node is not able to handle traffic / exchange any data with connected nodes. Not the other way around.

Again, the problem is not really the lack of resources on your node (CPU, drive speed), but rather the synchronization between the peer and db handling processes being screwed up and ending up with just timeouts preventing any data exchanges. Therefore, when you monitor your node, you don't see any strains, and yet it still is hosed.

Try to limit the number of peers to 10 and see how your nodes will hold during the next dust storm.

Another way to look why your 70% approach is wrong is to increase the number of peers to say 200 or so. Whether it is 80 or 200 peers, either number is just picked from a hat, but 200 is most likely going to strain your node. Potentially, with that 200 peers setting, you will start seeing those stalled nodes, even though there is no dust storm going on. That number 80 was picked up because the dev that was working on that code had his computer running smoothly on basically non-existent net space at that time, and that number got stuck, instead of being rejected during the code review as a broken solution.

[karstenbruch]

you say it is some "the synchronization between the peer and db handling processes being screwed up" or "but rather the chia code is bad".
Then the node would spew errors in the log. I had that when the networking in windows(one node is windows one is on ubuntu) just misteriously failed. Actually you do not really know what you are talking about but rather assume the networking code of the chia node is bad ? Maybe there is something that could be optimized. But i weathered 3 or more dustings just fine. If you are right it should have happenend all the time to all nodes. Well, it did not. Why is that ? I assume most of them run with the default. I just changed it to see if it would change anything.

Could be just some edge cases, maybe, but the traffic a dusting produces is almost not noticeable on my network. And its not that it is superior.. chia does not need much of a network. It was like a little more if nothing else is happening. So i don't know how both nodes came to a halt running without errors.

Event if it would be internal errors that do not reach the log. Those would probably lead to some memory leaking and i saw none of that.

So i do not think reducing the amount of connections would change anything.

I still think i would be an improvement when the node would keep an eye on the movement of its peers and act accordingly.

Oh and nothing got hosed. I was just a maximum amount of transactions.

And i did set my windows node to 120 to gather more ips for my geo-ip-map... did not make much of a difference.

[Jacek-ghub]

Let me try to follow your logic.

Just before the dust storm, your node had roughly 100% of synced peers. However, all those nodes "turned out" to be lagging ones as dust storm progressed. You assumed that out of 81 nodes (80 peers plus your node), the problem was with those 80 peers, not your node. Good luck with that line of thinking. (Again, the network had potentially 50%-70% or more nodes that were not affected at all, and your peers should reflect that.)

Once your node magically lost all those "good" peers, 100% of replacement peers were the lagging ones. Again, you assume 100% failure rate on other nodes, still clutching to the thought that your node was the only one that was working, and some magic happened, and those "bad" nodes were swarming your node. I guess, good luck with that line as well.

Yeah, everything was conspiring against your node, and when 100% of all other nodes turned bad, your node was the only one standing, but couldn't work anymore as there was not a single good node out there. Good luck with this as well.

Still, you want to have some number (70%) that would magically turn part of the "bad" peers your node has into "good" ones. However, you fail to see that you initially "lost" 100% of good nodes, and then you were getting 100% of "lagging" nodes as replacement. So, where that 70% of good nodes should be coming from?

So, what exactly that 70% should do? What magic that number should invoke to find good nodes, when there is only one good node - yours?

[brause]

I really do not know what you are trying to do here ... its not constructive to say the least. I only know what i saw on my nodes. Not what you are making it out to be.

I still think i would be an improvement when the node would keep an eye on the movement of its peers and act accordingly.