proposer: general task handling, timeouts, and failure modes #89
Description
In general the failure mode of the proposer seems overly drastic:
Within the tasks, there are a lot of conditions that raise an error and make the whole task fail, which then will get retried
once the game-state is evaluated again on the next processing loop tick and the same action is triggered internally.
Especially in something like the GameProving task, not having a retry mechanism and fail the whole task immediately means potentially throwing away all (even successful) requests to the prover-network, e.g. due to a temporary l1-client failure in eth_getProof.
There are failures that can't be recovered from, but even then we potentially don't want to end up in a retry loop of the whole proving task
and keep on spending gas on the prover network indefinitely.
Instead, we want calls that can recover to be wrapped in sensible, configureable timeout, retry, backoff loops.
A concrete feature request should be carefully determined prior.