Big Dataset Support

[matheus23]

Summary

Widening the possible use cases to include bigger data sets.

Problem / Need / Opportunity

Due to practical IPFS block size restrictions, WNFS can only safely encode up to 1000-2000 directory entries in public and private directories.

Success Scenario

It's possible to create huge directories! We enable use cases like on-boarding bigger data sets comparable to all of wikipedia, the linux kernel or a nix cache.

Children

ETA: 2023-09-30

Notes

wnfs-wg/spec#8

[RangerMauve]

Would be cool to talk about Prolly Trees around when this happens. We're hoping to have some tests figured out by april on prolly trees to see how the performance works.