Expire and republish records received via PUT_VALUE and ADD_PROVIDER

[raulk]

We:

1. Store such data permanently, thus deviating from Kademlia and creating weakness in our system for various reasons.
2. Don't rebalance/republish these records, allowing these records to disappear from the network as nodes leave and join.

See discussion in https://discuss.libp2p.io/t/dht-hourly-key-value-republish/93/4.

[vyzo]

this will be a problem for ipns, as the original publisher may be offline.
cc @Stebalien

[raulk]

Note that IPNS does republish the name record every 4 hours:

https://github.com/ipfs/go-ipfs/blob/42e191c0170a8e818c3302dab9b5afbf968df405/namesys/republisher/repub.go#L27

But the way I see it, this is to deal with network churn, not because the DHT peers are actively pruning the record. Even if the IPNS record carries the expiration, all the DHT sees is a []byte. Maybe via a Validator it validates a record is not expired, but this would happen only reactively: when processing a PUT_VALUE or a GET_VALUE request.

[dirkmc]

I want to make sure I understand the reason that Kademlia requires a node to republish its key/value pairs every hour. Is it so that as peers enter and leave the DHT the key/value is rebalanced onto the nodes closest to the key? If that's the case then it should make IPNS work better, because it will be more likely to find 16 copies of the key/value pair.

[raulk]

@dirkmc in addition to that, it helps keep storage footprint in check -- although by itself it's insufficient to deter DoS attacks exploiting that vector.

[Stebalien]

I want to make sure I understand the reason that Kademlia requires a node to republish its key/value pairs every hour.

The requirement comes from the fact that the network isn't exactly stable.

We should definitely be throwing away old records, we even make sure to record the time received so we can do this. The main issue is that we simply don't sweep the datastore.

[raulk]

@dirkmc if you're up for making a contribution, we can guide you through it ;-)

[dirkmc]

Sure, I can do so - I've been working on the Javascript DHT so I will do the same there

[jacobheun]

Should we also do the same for ADD_PROVIDER? It seems like it will suffer from the same issues if not republished.

[vyzo]

provider records are already collected after 24hs, and changing that would be a difficult change to deploy.