Skip to content

Commit ae29aed

Browse files
committed
Draft a roadmap.
These are just my current thoughts on what we could do with a bunch of ADLs, taking the idea of the code here now and running with it.
1 parent 7b13eee commit ae29aed

File tree

1 file changed

+88
-0
lines changed

1 file changed

+88
-0
lines changed

ROADMAP.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
Roadmap
2+
=======
3+
4+
The purpose of this repo is to offer ways to read and write UnixFSv1 structures using standard (go-ipld-prime) `ipld.Node` and `ipld.NodeAssembler`,
5+
and also to "path" over these structures in natural ways.
6+
7+
This will be done by producing several [ADL](https://github.com/ipld/docs/blob/master/docs/advanced-layouts.md)s.
8+
9+
- ADL to make files appear as `kind`==`bytes`, where interally layout=trickledag:
10+
- [ ] Readable
11+
- [ ] Writable
12+
- [ ] Chunker as a callback/interface
13+
- (may choose to implement this as mostly shared code with the other file ADL, where the internal layout is just a parameter; undecided)
14+
- ADL to make files appear as `kind`==`bytes`, where interally layout=balanceddag:
15+
- [ ] Readable
16+
- [ ] Writable
17+
- [ ] Chunker as a callback/interface
18+
- (may choose to implement this as mostly shared code with the other file ADL, where the internal layout is just a parameter; undecided)
19+
- ADL to make unixfsv1 unsharded directories look like a regular `kind`==`map`:
20+
- (e.g. just walk the list of links and select something based on the name field inside the list as if that was a map key.)
21+
- [x] Readable
22+
- [ ] Writable
23+
- ADL to traverse the HAMT which unixfsv1 uses for sharded directories as a `kind`==`map`:
24+
- [ ] Readable
25+
- [ ] Writable
26+
- (may be very similar to other HAMT libraries; may choose implement this in other repo with those, if more convenient/maintainable to do so; undecided)
27+
- (will probably use the above ADL for unsharded directories internally as necessary when reaching leaf nodes which have that format.)
28+
- ADL to traverse unixfsv1 directories as a `kind`==`map`, and land directly on the next file/directory (rather than on its metadata):
29+
- [ ] Readable
30+
- there will *not* be a writable side to this! Directories can't have entries added to them without metadata, so a symmetric write operation which is symmetric to this read operation is not defined.
31+
- (this is how natural pathing of the form `/ipfs/{cid}/xxx/yyy/zzz.ext` is served -- the `xxx` and `yyy` segments are served by lookups across this kind of ADL.)
32+
- (note how this and the above ADL _read the same substrate data_ -- while _interpreting it in different ways_.)
33+
34+
35+
### Why do this as ADLs?
36+
37+
Several reasons.
38+
39+
In general, ADLs mean we can use these things with "plain" IPLD concepts, and that provides a lot of value.
40+
41+
- IPLD standard features such as `traverse.*` functions work over ADLs transparently.
42+
- That includes IPLD Selectors. Selectors can just walk over an ADL map the same way they walk over a regular map.
43+
- That means even very advanced systems like Graphsync will work transparently underneath these structures...
44+
- *without* needing additional complex code to have a special understanding of any of unixfsv1 data topology.
45+
- *without* needing to understand what a user is doing with the synthesized view of the data.
46+
- When developing other filesystem-like systems in the future, this is a pattern of design that we can repeat!
47+
- Using this pattern doesn't guarantee total interoperability... but it makes it a lot more likely for it to naturally appear.
48+
49+
50+
51+
Non-goals
52+
---------
53+
54+
There are a couple of feature groups which are _not_ intended to be part of the code in this repository,
55+
either because they have a sufficiently distinctive scope that they live in other repos,
56+
or because they're simply misfeatures:
57+
58+
- This repo doesn't concern itself with data transfer strategies.
59+
- When using `Reify()`-style methods, loading of data is a non-issue because you've already done it.
60+
- When in other scenarios, loading and storing data is handled by configuring a `ipld.LinkSystem`
61+
(and more specifically, the `ipld.BlockReadOpener` and `ipld.BlockWriteOpener` function interfaces),
62+
as is normal for systems based on go-ipld-prime.
63+
- ADLs that handle multiple blocks of data internally will take an `ipld.LinkSystem` as a parameter.
64+
- This should pair well with other data transfer strategies, such as Graphsync, Bitswap, or who knows what --
65+
as long as it can provide `ipld.BlockReadOpener` and `ipld.BlockWriteOpener`, it should integrate without issue.
66+
- These ADLs do not provide a way to generate Selectors that describe their internals.
67+
- Instead: supposing that you want this in order to do something like use Graphsync to request and transfer unixfsv1 data:
68+
you can accomplish this by using Selectors *on top of* the ADLs,
69+
and having your remote party agree to use the same ADLs in the same places as you both walk the data.
70+
- Mapping to other filesystems. (Whether that be the OS filesystem, or tar formats, or etc.)
71+
- These are valuable features, but not included in this repo.
72+
73+
74+
75+
Replaces
76+
--------
77+
78+
This repo should effectively replace several other repos:
79+
80+
- `ipfs/go-unixfs`
81+
- `ipfs/go-path`
82+
- ... and probably several others.
83+
84+
This is a *generational* approach to code.
85+
That is: this repo will provide semiotically comparable features to the repos it is replacing,
86+
but this repo will *not* replace *every* feature in those repos,
87+
and definitely will not contain exact replicas of those APIs.
88+

0 commit comments

Comments
 (0)