What is wrong with the blockchain since ~2021-12-20 11:00 UTC / new storm? #9637
Replies: 9 comments 4 replies
-
|
Hi, same problem here in 1.2.11 on rasp, 100% stale partials ! |
Beta Was this translation helpful? Give feedback.
-
|
whatever this is, my pi is dying because of that. Is this a new storm? Usually I am pro chia, but at those moments, it is hard. Everything feels like a big alpha version. |
Beta Was this translation helpful? Give feedback.
-
|
that's the reason i think |
Beta Was this translation helpful? Give feedback.
-
|
Just to let you all know...I run my chia on a high end server and I have experienced the same issues today. |
Beta Was this translation helpful? Give feedback.
-
|
Thats good to know. I have only a tiny farm with ~250 TiB. 2 hours ago, I had a look into the top ones of space pool and some of them also have issues. My farm is dead, completely stopped syncing. Maybe its because of the inbound connection limit. To me it looks like another storm. 2021-12-20T22:40:40.760 full_node full_node_server : INFO Not accepting inbound connection: {'host': '10.0.0.2', 'port': 8444}.Inbound limit reached. |
Beta Was this translation helpful? Give feedback.
-
|
I have many zombie(?) connections
The 10.0.0.2 is because of docker swarm. That's ok.FULL_NODE 10.0.0.2 53421/8444 e6032b2c... Dec 20 23:17:02 0.0|1.1 |
Beta Was this translation helpful? Give feedback.
-
|
As I have tried to explain in the other thread, having so many peers with height 0 implies that the problem is not with those peers, but rather with your node. Basically, your node is being overwhelmed with processing transactions, and all synchronization between peer and db handling processes is broken. The end result is that the node is able to connect but cannot do a full handshake to get the other nodes heights, not to mention get data from them. Adding insult to the injury, the code is still going through some worthless motions trying to do something about it, basically putting additional strain on the CPU / synchronization process. The fact that you may still see some nodes that have "good" heights is that those are "old" nodes with their counters stuck at at the point when the last data exchange happened with them, but I would assume that those peers are also dead for your node. So, the first thing to do is to check CPU utilization and start working on reducing the load. I also don't think that any extra button to speed up deletion of nodes should be added. The problem is mostly with the networking code, as it depends on some garbage magical numbers (e.g., 80 peers), and it sticks to those numbers, instead of adjusting the number of peers dynamically depending on the network / node condition. It is really retarded to have 300k people manually doing those changes, instead of having one engineer fix that crap. However, I think that the code should also have a shorter timeout to kick off nodes that cannot fully handshake / sync (those sitting idle with Up/Down MB not moving for a minute or two). Potentially adding another longer timeout before a new node may be accepted (just that would drive the number of connections down to a working number - a primitive back-off mechanism). Lastly, on my box (Intel i5), only one core was hosed / only one start_full_node process was maxing it out. The other cores / start_full_node processes were running more or less idle all the time. That implies that the offending code is single-threaded and cannot utilize CPU properly. So, all talks about getting a better H/W are kind of misguided. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, maybe there is some unused potential in the network code, especially in the initiation of connections. Or one of the system requirements should be a cpu with high single core performance. But chia will be "green" so one main focus should always be on In this case for the RPi 4/4GB more is not always more. The reduction of the peer limit was very successful - for me. After one day it seems that I have another positive side effect. The fluctuation of the estimiated plot size in space pool is really steady now. Yesterday a limit of 10 worked quiet well within the "storm" / full mem pool. Maybe I reduce it further to 7 or 5 to be safe. |
Beta Was this translation helpful? Give feedback.
-
|
I would really not go below 10. The lower you go, the more likely you may end up with just "slow" peers. Again, from p2p topology point of view, 3 is the hard limit. Of course, there is no magic in 10, but that seems to be like a good balance - and of course what is my good balance doesn't have to be yours :) |
Beta Was this translation helpful? Give feedback.
-
|
Having the same issue today on a high end server with port 8444 forwarded. |
Beta Was this translation helpful? Give feedback.
-
|
decreasing the Thx to @Jacek-ghub |
Beta Was this translation helpful? Give feedback.
-
|
Hi folks So there is another transaction storm off and on the last few days, causing lower end systems, especially Pis, to experience pain. here a few a things you can do with an RPi specifically to help:
Also I want to take a quick second to reiterate the minimum spec for a RPi: That;s a Pi4, with 4GB of ram, and an external (non SD card) for you blockchain DB. (And even then, an ideally "Fast" SD card for the OS layer, unless you run your OS on the external as well). |
Beta Was this translation helpful? Give feedback.
-
|
Short update after more than a week. Today, the mempool was full again, approx. between block 1339986 and 1341531. The PI 4 did a great job, no problems. The property Maybe the connection handling can be reviewed/improved in the future... just another task for the backlog ;) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What happened?
My RPI4 uses a custom docker image of chia. It works great, normally. Today, there are some strange logs and RPIs' cpu is under heavy load. I also have nearly 100% stale partials (probably because of the cpu load).
See logs below, they are flooded with messages
Version
1.2.11
What platform are you using?
Pi4
What ui mode are you using?
CLI
Relevant log output
Beta Was this translation helpful? Give feedback.
All reactions