Sync procedure is fundamentally broken

[antst]

You can see that there is a number of sync bugs and discussions. And they are appearing with increasing rate.
I looked into it for last 2 days and outcome is simple. Once number of transactions increased, sync speed is lagging speed of grows of blockchain.

In my test which goes for last 36 hours which I started from scratch, I can't catch current block, and this is happening on hardware you can't blame: threadripper, very SSD for DB, real 1GbE connection.
This means once typical node got out of sync for whatever reasons, it is difficult to catch up. And this should be reason of this massive surge of out of sync issues.

[markusmazurczak]

What out of sync timeframe you are talking about? I am able to catch up after beeing out of sync for 6 hours easily with an old i5 processor and an low budget SSD

[antst]

To me it looks it depends on your "luck" and peers you are connected to before you reached limit of peers.

[antst]

this is example from farmer node (no harvesters)

Mon 17 May 2021 09:38:18 AM UTC Block Height: 275398
Mon 17 May 2021 09:42:49 AM UTC Block Height: 275405

47 seconds per block.

[markusmazurczak]

Ja, that sound legit. Take a look into #3298. I am currently trying to hard-ban peers that I can get no connection to so that the garbage collector can remove them from ChiaServer's internal peer-list

[antst]

Also, chia_full_node is bounded by CPU, it shows 100% CPU usage, while in this terrible sync.

[antst]

Another thing I see from logs, I am feeding known blocks to other nodes with speed of light, but this also, probably, one of reasons at CPU bounding.

[leefarg]

I echo this same problem - got out of sync a few days ago, and unable to regain sync. After restart multiple times and complete db delete, in 24 hours, I am only a little over half-sync'd.

[antst]

One more thing, looks like sometimes it just getting stuck on particular block, I presume, waiting something from particular peer and not getting

[tojefest]

Ja, that sound legit. Take a look into #3298. I am currently trying to hard-ban peers that I can get no connection to so that the garbage collector can remove them from ChiaServer's internal peer-list

at this moment this solution work... I think this only peers problem. If i have luck, sync work great, bad peer and not synced for 6hours. Synchronization is broken, totally. This isn`t problem of db, cpu or ssd, network. Only app have big problem. Manually wallet restart can helps but it afffect farming...

[antst]

Can you point towards procedure?

[tojefest]

i work on it...
at this moment i use simply powershell script:

this is sample :)
loop
.\chia start wallet -r
.\chia.exe show --add-connection lucky_peer_ip:8444
.\chia.exe show --add-connection lucky_peer_ip:8444
.\chia.exe show --add-connection node-eu.chia.net:8444
ping -n 960 gatewayip
reloop

App looks like use only first added peer, but that is bad solution for farming.

[antst]

i work on it...
at this moment i use simply powershell script:

this is sample :)
loop
.\chia start wallet -r
.\chia.exe show --add-connection lucky_peer_ip:8444
.\chia.exe show --add-connection lucky_peer_ip:8444
.\chia.exe show --add-connection node-eu.chia.net:8444
ping -n 960 gatewayip
reloop

App looks like use only first added peer, but that is bad solution for farming.

What speed of sync you have in this case? number of seconds per block

[tojefest]

40s per block as long as i`m lucky
just like others:
ValueError: Error short batch syncing, invalid/no response for 293426-293458
full_node full_node_server : WARNING Banning 31.30.70.129 for 10 seconds+
full_node asyncio : ERROR Task exception was never retrieved

Only wallet restart helps for me.
Fully resync new db: 30h to 42h.

[antst]

40s per block as long as i`m lucky

it is still slower than chain grows.