contractcourt+channeldb: implement garbage collection for the WitnessCache

[Roasbeef]

Currently, the ChannelLink and the channelArbitrator leverage an interface (the WitnessBeacon and its concrete implementation the preimageBeacon) in order to notify distinct links when we learn of a new preimage off-chain, or extract a pre-image on chain due to a force close then sweep by the remote party. Atm, the set of stored preimages on disk is never garbage collected. As a result, the storage (though just 32 bytes for each payment) will grow linearly as more payments are forwarded by routing nodes. This isn't an issue now, or in the mid-term, but for longer lived nodes that forward millions of payments over their lifetime, the space may start to become an issue. In order to resolve this, an eviction methodology should be developed for the preimage cache.

• Extend the default preimageBeacon struct (and possibly the underlying channeldb.WitnessCache to support automatic garbage collection of the on disk witnesses. We'll need to strike a balance between retaining them long enough for a link that received an incoming HTLC to pull the HTLC on chain after we learn of the preimage from the outgoing link off-chain, and the duration of time that we'll store each of the witnesses.

[Crypt-iQ]

Hey is anybody working on this one?

[halseth]

@Crypt-iQ Up for grabs! 😀

[Crypt-iQ]

Ok! Will take this one

[saubyk]

Converting this to a discussion. Can be reverted to the issue or a new one opened up if any development is planned for this.