History expiry

[martintomazic]

Long term solution to the growing state problem.

Context

As we are getting further away from the genesis, keeping all historical state becomes impractical, thus nodes (especially validators) start pruning (oasisprotocol/docs#1526).

However, as the nodes start pruning we are risking data availability issues (at least for the genesis sync).

Currently, the network still has nodes that serve historical data and we also offer snapshots. Nevertheless, there will be less and less such nodes, which decreases data availability and increases the risk of (accidentally) loosing it.

Proposed solution

History Expiry.

Let's define a sane default, for which all network participants (except for stateless clients) guarantee, it is possible to sycn from. I believe 1 year would be a good value.

If the network is always capable of serving a State DB checkpoint that is one year old, and if by default nodes always preserve 1 year of blocks and or storage diffs, then our network guarantees history expiry invariant.

Implementation space:

History Expiry Checkpoint

In addition to existing checkpoints, there will be another historical checkpoint created and replaced every month (target age of history expiry). In practice it will replace the current genesis checkpoint. I.e. every node serving checkpoints will fetch, store and serve it, even if they only preserve one month of state.

Pruning

Operator would still be able to prune runtime and consensus state to any value it wants, however we will not allow pruning of consensus blocks and or runtime storage diffs younger then expired history.

[peternose]

will not allow pruning of consensus blocks and or runtime storage diffs younger then expired history.

Could you also provide the expected disk usage for historical checkpoint, consensus blocks and runtime storage diffs for one year and one runtime? And how much will one node gain if he prunes runtime and consensus state to e.g. few blocks vs whole year.