Skip to content

v3.3.0 - Add Sharded Store Support

Latest

Choose a tag to compare

@Faolain Faolain released this 26 Sep 06:08
v3.3.0
5d7b6e9

Previously as described by the very name of the library itself, you could access a zarr on ipfs using a HAMT data structure. This meant that the larger the dataset, the more likely it would be you would have to make multiple round-trip calls in order to traverse the tree to get the content you requested (depending on the width of the HAMT that was initiated - the narrower in width the deeper more clashing keys would have to go). The ShardedStore solves this with some slight tradeoffs where a manifest instead stores the CIDs of chunks in flat cbor lists which the sharded store accesses via byte offsets.

  • CLI hamt -> shardedstore converter included