Retry Mechanism for Hyperlane Messages #296
Description
Currently if a Hyperlane snapshot is stored as unfinalized and the proof submission fails, the snapshot will never be picked up again and all intermediary messages are lost.
In the next iteration the prover will continue with a new unfinalized snapshot after inserts and prove the newly occurred messages.
So in practice this should never break, unless invalid transactions are included in the tree and therefore an invalid snapshot is built, but it's still not perfect because in theory messages can get lost / skipped. This has not happened and is purely theoretical, but a network error on proof submission could cause this loss.
We should occasionally check for unfinalized snapshots at height < LATEST-1, note that LATEST-1 should always be unfinalized but all snapshots before that should always be finalized.
Implementation high-level:
On service startup it checks for the first unfinalized snapshot in range last_finalized..height-2 (because height - 1 is expected to be unfinalized / that's fine) and generates a proof.
This way it could happen that messages are proven twice, but I think that our design allows for that.