|
| 1 | +Roadmap |
| 2 | +======= |
| 3 | + |
| 4 | +The purpose of this repo is to offer ways to read and write UnixFSv1 structures using standard (go-ipld-prime) `ipld.Node` and `ipld.NodeAssembler`, |
| 5 | +and also to "path" over these structures in natural ways. |
| 6 | + |
| 7 | +This will be done by producing several [ADL](https://github.com/ipld/docs/blob/master/docs/advanced-layouts.md)s. |
| 8 | + |
| 9 | +- ADL to make files appear as `kind`==`bytes`, where interally layout=trickledag: |
| 10 | + - [ ] Readable |
| 11 | + - [ ] Writable |
| 12 | + - [ ] Chunker as a callback/interface |
| 13 | + - (may choose to implement this as mostly shared code with the other file ADL, where the internal layout is just a parameter; undecided) |
| 14 | +- ADL to make files appear as `kind`==`bytes`, where interally layout=balanceddag: |
| 15 | + - [ ] Readable |
| 16 | + - [ ] Writable |
| 17 | + - [ ] Chunker as a callback/interface |
| 18 | + - (may choose to implement this as mostly shared code with the other file ADL, where the internal layout is just a parameter; undecided) |
| 19 | +- ADL to make unixfsv1 unsharded directories look like a regular `kind`==`map`: |
| 20 | + - (e.g. just walk the list of links and select something based on the name field inside the list as if that was a map key.) |
| 21 | + - [x] Readable |
| 22 | + - [ ] Writable |
| 23 | +- ADL to traverse the HAMT which unixfsv1 uses for sharded directories as a `kind`==`map`: |
| 24 | + - [ ] Readable |
| 25 | + - [ ] Writable |
| 26 | + - (may be very similar to other HAMT libraries; may choose implement this in other repo with those, if more convenient/maintainable to do so; undecided) |
| 27 | + - (will probably use the above ADL for unsharded directories internally as necessary when reaching leaf nodes which have that format.) |
| 28 | +- ADL to traverse unixfsv1 directories as a `kind`==`map`, and land directly on the next file/directory (rather than on its metadata): |
| 29 | + - [ ] Readable |
| 30 | + - there will *not* be a writable side to this! Directories can't have entries added to them without metadata, so a symmetric write operation which is symmetric to this read operation is not defined. |
| 31 | + - (this is how natural pathing of the form `/ipfs/{cid}/xxx/yyy/zzz.ext` is served -- the `xxx` and `yyy` segments are served by lookups across this kind of ADL.) |
| 32 | + - (note how this and the above ADL _read the same substrate data_ -- while _interpreting it in different ways_.) |
| 33 | + |
| 34 | + |
| 35 | +### Why do this as ADLs? |
| 36 | + |
| 37 | +Several reasons. |
| 38 | + |
| 39 | +In general, ADLs mean we can use these things with "plain" IPLD concepts, and that provides a lot of value. |
| 40 | + |
| 41 | +- IPLD standard features such as `traverse.*` functions work over ADLs transparently. |
| 42 | + - That includes IPLD Selectors. Selectors can just walk over an ADL map the same way they walk over a regular map. |
| 43 | +- That means even very advanced systems like Graphsync will work transparently underneath these structures... |
| 44 | + - *without* needing additional complex code to have a special understanding of any of unixfsv1 data topology. |
| 45 | + - *without* needing to understand what a user is doing with the synthesized view of the data. |
| 46 | +- When developing other filesystem-like systems in the future, this is a pattern of design that we can repeat! |
| 47 | + - Using this pattern doesn't guarantee total interoperability... but it makes it a lot more likely for it to naturally appear. |
| 48 | + |
| 49 | + |
| 50 | + |
| 51 | +Non-goals |
| 52 | +--------- |
| 53 | + |
| 54 | +There are a couple of feature groups which are _not_ intended to be part of the code in this repository, |
| 55 | +either because they have a sufficiently distinctive scope that they live in other repos, |
| 56 | +or because they're simply misfeatures: |
| 57 | + |
| 58 | +- This repo doesn't concern itself with data transfer strategies. |
| 59 | + - When using `Reify()`-style methods, loading of data is a non-issue because you've already done it. |
| 60 | + - When in other scenarios, loading and storing data is handled by configuring a `ipld.LinkSystem` |
| 61 | + (and more specifically, the `ipld.BlockReadOpener` and `ipld.BlockWriteOpener` function interfaces), |
| 62 | + as is normal for systems based on go-ipld-prime. |
| 63 | + - ADLs that handle multiple blocks of data internally will take an `ipld.LinkSystem` as a parameter. |
| 64 | + - This should pair well with other data transfer strategies, such as Graphsync, Bitswap, or who knows what -- |
| 65 | + as long as it can provide `ipld.BlockReadOpener` and `ipld.BlockWriteOpener`, it should integrate without issue. |
| 66 | +- These ADLs do not provide a way to generate Selectors that describe their internals. |
| 67 | + - Instead: supposing that you want this in order to do something like use Graphsync to request and transfer unixfsv1 data: |
| 68 | + you can accomplish this by using Selectors *on top of* the ADLs, |
| 69 | + and having your remote party agree to use the same ADLs in the same places as you both walk the data. |
| 70 | +- Mapping to other filesystems. (Whether that be the OS filesystem, or tar formats, or etc.) |
| 71 | + - These are valuable features, but not included in this repo. |
| 72 | + |
| 73 | + |
| 74 | + |
| 75 | +Replaces |
| 76 | +-------- |
| 77 | + |
| 78 | +This repo should effectively replace several other repos: |
| 79 | + |
| 80 | +- `ipfs/go-unixfs` |
| 81 | +- `ipfs/go-path` |
| 82 | +- ... and probably several others. |
| 83 | + |
| 84 | +This is a *generational* approach to code. |
| 85 | +That is: this repo will provide semiotically comparable features to the repos it is replacing, |
| 86 | +but this repo will *not* replace *every* feature in those repos, |
| 87 | +and definitely will not contain exact replicas of those APIs. |
| 88 | + |
0 commit comments