Rasp Pi 4 GB Sync from fork point failed err: database disk image is malformed[Bug]

[NitsugaCodu]

What happened?

Dear All,

The database malformed 3 times in less than a month and i dont know what is going on as i didnt change anything.

Model: Rasperry Pi 4 - 4GB / Running Farmer (Chia start farmer)
"High end" microsd = 128 GB (Database is around 40 GB)

I rebooted, shutdown, stop all -d, etc but i didnt stop using kill -9

Thanks for your support.

Version

1.2.11

What platform are you using?

Pi4

What ui mode are you using?

CLI

Relevant log output

File "/home/XXXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: [2a01:e0a:a38:8cd0:aaa1:59ff:fe8b:8977]

2022-01-18T21:46:31.017 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:46:31.827 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:47:28.984 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/XXXXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 62.43.149.199

2022-01-18T21:48:02.663 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/XXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 66.147.239.139

2022-01-18T21:51:43.049 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:51:44.124 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:53:36.559 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T21:55:21.241 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/XXXXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 184.98.231.187

2022-01-18T21:56:54.320 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:56:55.731 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T21:58:50.507 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:00:57.044 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/XXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 171.96.101.22

2022-01-18T22:02:08.205 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:02:08.945 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:06:39.173 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:07:20.439 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:07:21.211 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:11:18.260 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:12:31.556 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:12:32.348 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:13:34.076 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'RuntimeError'>Traceback (most recent call last):
  File "/home/XXXXX/chia-blockchain/chia/full_node/full_node.py", line 807, in _sync
    raise RuntimeError(f"Weight proof did not arrive in time from peer: {weight_proof_peer.peer_host}")
RuntimeError: Weight proof did not arrive in time from peer: 141.94.111.8

2022-01-18T22:15:44.572 full_node full_node_server        : WARNING  Cannot write to closing transport 121.29.69.112
2022-01-18T22:17:43.393 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:17:44.431 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space

2022-01-18T22:17:43.393 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:17:44.431 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:18:20.750 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:22:54.640 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:22:55.933 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
sync from fork point failed err: database disk image is malformed
2022-01-18T22:23:40.990 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:26:25.486 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:28:07.669 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:28:08.452 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:30:37.750 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:33:18.462 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:33:19.256 farmer chia.farmer.farmer         : ERROR    Failed to update farmer information on the pool https://na1.pool.space
2022-01-18T22:34:47.491 full_node chia.full_node.full_node: ERROR    sync from fork point failed err: database disk image is malformed
2022-01-18T22:37:08.063 full_node chia.full_node.full_node: WARNING  Syncing failed, CancelledError

[emlowe]

SD cards are not able to run the DB because there are too many I/O requests - which will swamp the SD card small over-provisioning and lack of support for TRIM. This can easily result in extremely poor performance for any SD card. I can't say for sure this would also cause any DB corruption - but the end result is that the SD card is not a recommended place for the DB.
Database corruption can only be resolved by resyncing, and it is highly recommended you use any regular SSD over USB for the DB.

One other item to check is if you have any setting for db_sync in config.yaml

[NitsugaCodu]

About SD, its strange because i have another Pi without any problem. Only difference is the 8GB and perhaps is not swaping but its strange because it work for days/week until it get wrong

This is the info i found about sync but i didnt find any db_sync in config yaml.

How do i change that chia folder is on an external device? CLI Ubuntu?

wallet:
database_path: wallet/db/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
enable_profiler: false
full_node_peer:
host: localhost
port: 8444
initial_num_public_keys: 100
initial_num_public_keys_new_wallet: 5
introducer_peer:
host: introducer.chia.net
port: 8444
logging: *id001
network_overrides: *id002
num_sync_batches: 50
peer_connect_interval: 60
port: 8449
recent_peer_threshold: 6000
rpc_port: 9256
selected_network: mainnet
short_sync_blocks_behind_threshold: 20

short_sync_blocks_behind_threshold: 20
wallet:
database_path: wallet/db/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
enable_profiler: false
full_node_peer:
host: localhost
port: 8444
initial_num_public_keys: 100
initial_num_public_keys_new_wallet: 5
introducer_peer:
host: introducer.chia.net
port: 8444
logging: *id001
network_overrides: *id002
num_sync_batches: 50

[emlowe]

There are multiple ways to change the directory - one of the easiest is to just symlink the entire ~/.chia/mainnet directory to your external SSD drive. However, you can also symlink just the DBs or DB directories - or you can edit the path in config.yaml.

If you are already synced, then often the SD card might be able to stay synced. But the syncing from 0 today results in a large amount of I/O to the drive which most SD cards cannot handle.