Node crashes after a few weeks

[dannywillems]

From a user on Discord:

Right now v0.16.0 seems to be the most stable but does crash after few weeks

They use the Docker image. We asked them to provide the logs.
Assigning to myself for now to clarify the task and get logs.

[tizoc]

@dannywillems the usual situation is that the node fails to produce a block (or more likely prove it), and it has no recovery mechanism, so it just panics. Other than fixing the bug, this would provide a way for the node to continue running without losing track of the error https://github.com/o1-labs/openmina/pull/1097

[tizoc]

Two other related issues (rare but I think it still happens, these sync calls are high priority to fix anyway):

• https://github.com/o1-labs/openmina/issues/43 (mostly all fixed except for the transaction pool one from the next issue)
• https://github.com/o1-labs/openmina/issues/994

because of the synchronous calls (something should never happen in properly implemented state machines) to fetch the accounts from the ledger, rarely some race condition seems to be hit which crashes the node.