ERROR Exception: Error short batch syncing, could not fetch block

[DDMurr]

2021-09-01T20:13:04.497 full_node full_node_server : ERROR Exception: Error short batch syncing, could not fetch block at height 800024 <class 'ValueError'>, closing connection {'host': '188.187.42.11', 'port': 16664}. Traceback (most recent call last):
File "chia\server\server.py", line 563, in api_call
File "asyncio\tasks.py", line 442, in wait_for
File "chia\server\server.py", line 560, in wrapped_coroutine
File "chia\server\server.py", line 553, in wrapped_coroutine
File "chia\full_node\full_node_api.py", line 106, in new_peak
File "chia\full_node\full_node.py", line 411, in new_peak
File "chia\full_node\full_node.py", line 247, in short_sync_batch
ValueError: Error short batch syncing, could not fetch block at height 800024

[DDMurr]

I continue to get this error message. Do I need to do something to correct it or not worry about it?

2021-09-09T19:21:59.376 full_node full_node_server : ERROR Exception: Error short batch syncing, could not fetch block at height 836993 <class 'ValueError'>, closing connection {'host': '98.249.137.155', 'port': 16664}. Traceback (most recent call last):
File "chia\server\server.py", line 563, in api_call
File "asyncio\tasks.py", line 442, in wait_for
File "chia\server\server.py", line 560, in wrapped_coroutine
File "chia\server\server.py", line 553, in wrapped_coroutine
File "chia\full_node\full_node_api.py", line 106, in new_peak
File "chia\full_node\full_node.py", line 411, in new_peak
File "chia\full_node\full_node.py", line 247, in short_sync_batch
ValueError: Error short batch syncing, could not fetch block at height 836993

[stikeleather]

I'm getting this error message now after updating to the most recent version (1.2.6)

[loppefaaret]

every now and then, an error happens, it can be the peer that you have requested data from crashed, lost internet, or just plain closed down chia - you will see this on your side - as long as it is a "short sync" error - your node is relative close to being on the peak height of the chain - it will try and get the block from another peer...

[Jacek-ghub]

@loppefaaret

None of the conditions your listed is actually an error (peer crashed, ...), as the requesting node is happy doing what it should be doing. Adding to that that the node still has other peers to ask for the data, makes it even less of an error, just a warning while going through the data request loop. Once the loop is exhausted, it may be an error with the local setup (connection down, ...). At this point a different error should be kicked off stating that the node is potentially isolated.

The fact that we see a Traceback output there means that the code doesn't handle that case at all, and barfs instead of smoothly moving on to the next peer.

Maybe that ERROR should be reclassified to be above INFO level, as it is clearly only debugging info for the engineering/QA teams, and has no value for in the field node owner. It just pollutes logs, and makes end users be less sensitive to warning/errors they see.

[loppefaaret]

@Jacek-ghub I'm not entirely sure how you mean it isn't an error ?
Let me pull up a debug.log from this week, and show you an example of what a full_node reports when it looses contact with a peer it is trying to sync from:

2021-10-01T12:51:33.459 full_node chia.full_node.full_node: INFO     Requesting weight proof from peer 37.45.242.161 up to height 936950
2021-10-01T12:53:00.970 full_node chia.full_node.weight_proof: ERROR    failed validating weight proof recent blocks
2021-10-01T12:53:01.064 full_node full_node_server        : INFO     Connection closed: 37.45.242.161, node id: 30376825e1315e765cd22ede36e0f533dee257a2d8d34a8d52312b1c794a4ce8
2021-10-01T12:53:01.065 full_node full_node_server        : WARNING  Banning 37.45.242.161 for 600 seconds
2021-10-01T12:53:01.066 full_node chia.full_node.full_node: INFO     peer disconnected {'host': '37.45.242.161', 'port': 8444}
2021-10-01T12:53:01.066 full_node chia.full_node.full_node: ERROR    Error with syncing: <class 'ValueError'>Traceback (most recent call last):
  File "chia\full_node\full_node.py", line 741, in _sync
ValueError: Weight proof validation failed

2021-10-01T12:53:01.067 full_node chia.full_node.full_node: INFO     long sync done

Thats an example of a sync attempt not succeeding, although in another flavor than the one DDMurr reported above (this is an INFO level log from 1.2.8). As you can see, it does end up with ERROR entries from the full_node.
I must be misunderstanding what error I'm referring to above, that isn't an actual error ?

[Jacek-ghub]

@loppefaaret

"every now and then, an error happens, it can be the peer that you have requested data from crashed, lost internet, or just plain closed down chia" Agreed that errors happen no questions about that, but not every condition is an error. The fact that "the peer that you have requested data from crashed" is that peer problem/error, not really mine, as my node is still doing fine, and have other peers to work with.

P2P protocol is based on flaky peers, as such one peer disappearing is just a part of a standard procedure, and you move on trying another peer. Only once you have exhausted the list of peers and still have no data you were looking for (i.e., from the local node point of view, all peers have irrecoverable issues), this becomes a mine error/problem, as it suggest that the local node has issues (ISP, router, Ethernet, OS, ...), and an error is needed to mitigate those issues.

Going with your notion that somehow it is still an error. What the local node can do in such case? What the user can do to react to such error? There is no way to contact such remote node as it is more or less hidden who is behind that connection (as you said such peer could be just shut down, crashed, lost connection)? There is nothing that the local node can do, but as P2P specifies, it should quietly move to the next peer, and just request the same data.

I would also check why there is that Traceback there. Why that code spits out exception. If that code would be handling the errors via catching those exceptions, then as above, it should be quiet, as it is a well know condition, so no need to issue errors/warnings/... This is apparently not the case, and suggests that the offending code basically barfs there.

One more take on this error is that such condition could be considered as error if:

1. the local code is not prepared to handle it (surprised by that condition)
2. the condition causes instability in normal behavior

Looks like here we have #1 case, as the system is running smoothly. It is also clearly not #2 condition, as we are told that this is not an issue at all.

If you check what code assertiveness means, is you log issues that you don't like (but you never call them errors), and are out of your control, as it helps to focus on the code parts that potentially lead to such condition (and you are off the hook for such problems). If on the other hand you just spit messages restating that the code is not properly handling some issues, we still don't know where the source of those errors are, as the main focus is on the offending code. That also implies that such case could be handled, to at least mitigate the problem, and therefore no need for such errors.

Also, saying that this is an error, because it is in the log is a bit circular, isn't it? Looking at that section, I really don't understand whey the fourth line is an error. There is no error condition there, the code just decided to ban that peer, and is moving on to the next one. Shouldn't that be an INFO level message? The same with the sixth line, it is just INFO not-syncing condition, and the code moves to "long sync done" state, where you could inform that either it was not successful, and the next node will be tried. Of if that was the end of the loop, that should say that the loop got exhausted, and no data was received, and that would warrant ERROR condition, as most likely the problem is local.

One more thing. The code that you have shown is a clean code. Just logged messages. On the other hand, the OP code has Trackbacks, as such that code just barfed there, and whichever part caught that exception is trying to recover, and is also spitting that error without the understanding of what condition was really the issue (absolutely, this is an error on this level, but not on the one that barfed). You can see how many levels that code that caught that exception is away for the offending one.

[BrandtH22]

@DDMurr , the newest version of chia has fixes in place to resolve this issue. Please download it from here: https://www.chia.net/downloads/

Since this issue has been resolved with the latest version(s) of chia we will be closing this ticket but if we have closed this ticket in error do not hesitate to reach out to us again with any followup questions, comments, or if the issue persists after an update.

Please note that upgrading major versions of chia generally requires the config.yaml to be recreated using these steps:
1. Stop chia
2. Rename the config or move it to a different directory
3. In CLI run chia init
4. Copy any custom settings from the old config to the new (plot drive directories, pool info, trusted node info, etc) - be careful here as yaml is very picky about formatting including those leading spaces
5. Start Chia

The best place to reach our support team is on Discord (https://discord.gg/chia) or by reopening this ticket.