Skip to content

Commit 0fcfff1

Browse files
StebalienStebalien
committed
propose downloading go-ipfs migrations over IPFS
1 parent 69d4ffa commit 0fcfff1

File tree

1 file changed

+152
-0
lines changed

1 file changed

+152
-0
lines changed

‎proposals/migrate-over-ipfs.md

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Download migrations over IPFS
2+
3+
Authors: Stebalien
4+
5+
Initial PR: TBD <!-- Reference the PR first proposing this document. Oooh, self-reference! -->
6+
7+
## Purpose &amp; impact
8+
#### Background &amp; intent
9+
_Describe the desired state of the world after this project? Why does that matter?_
10+
11+
Currently, when the on-disk layout of the datastore changes, go-ipfs will download a "migration"
12+
tool over HTTPS as-needed to "upgrade" the datastore. Unfortunately, this happens over HTTPS so:
13+
14+
1. We're not dogfooding our own tech.
15+
2. It doesn't work in places where https://dist.ipfs.io is blocked (e.g., China).
16+
17+
At the moment, we're unable to ask Brave to update go-ipfs to the latest version because the latest
18+
version will need to download one of these migrations and many of Brave's users are in China.
19+
20+
#### Assumptions &amp; hypotheses
21+
_What must be true for this project to matter?_
22+
23+
- IPFS in Brave needs to matter.
24+
- A significant number of Brave users need to be in China.
25+
- Fetching migrations over IPFS needs to be reliable.
26+
27+
#### User workflow example
28+
_How would a developer or user use this new capability?_
29+
30+
- When starting the go-ipfs daemon with the `--migrate` flag, the migrations would be fetched over IPFS instead of HTTPs.
31+
- When starting the go-ipfs daemon in a bundled application like IPFS Desktop or Brave, the user shouldn't notice anything except that we make no connections to https://ipfs.io when starting go-ipfs the first time after upgrading.
32+
33+
#### Impact
34+
_How would this directly contribute to web3 dev stack product-market fit?_
35+
36+
- This would unblock updating go-ipfs in Brave, allowing us to ship bug fixes and new features to Brave users.
37+
- This would prove out IPFS as a way to ship code/updates to users.
38+
39+
#### Leverage
40+
_How much would nailing this project improve our knowledge and ability to execute future projects?_
41+
42+
This is ones step towards our goal of updating go-ipfs itself over IPFS. In terms of knowledge, not much.
43+
44+
#### Confidence
45+
_How sure are we that this impact would be realized? Label from [this scale](https://medium.com/@nimay/inside-product-introduction-to-feature-priority-using-ice-impact-confidence-ease-and-gist-5180434e5b15)_.
46+
47+
10? We're not shipping a new release to brave until we have _a_ way to ship repo migrations to users in China.
48+
49+
## Project definition
50+
51+
#### Brief plan of attack
52+
53+
**Design sketch:**
54+
55+
When migrating a repo:
56+
57+
1. Create a new _temporary_ repo (go-ipfs can't read the current repo because it's an older version).
58+
a. If the local go-ipfs node uses a swarm key, skip to step 3.a (download over HTTPs).
59+
2. Start a new _temporary_ go-ipfs node in the temporary repo.
60+
a. This node should not listen for inbound connections as it has no way to know which ports/transports should be configured (can't read the config).
61+
b. This node should not expose an API/gateway.
62+
3. Download the required migration binaries using the temporary go-ipfs node.
63+
a. If this fails, download over HTTPs.
64+
4. Migrate the main go-ipfs node's repo.
65+
5. Start the main go-ipfs node and _transfer_ (but don't pin) the downloaded migrations into the main repo so that others can download it from this node.
66+
6. Shut down the temporary node and delete the temporary repo.
67+
68+
Provide the following configuration section:
69+
70+
```js
71+
{
72+
"Migration": {
73+
// When true, always use IPFS. When false, never use IPFS. When unset, pick the default
74+
// behavior.
75+
"UseIPFS": true|false,
76+
// When true or unset, use the default gateway. When false, don't use a gateway. When a
77+
// string, use the specified gateway.
78+
"UseGateway": true|false|"my-gateway",
79+
// Whether or not to keep the migration after downloading it.
80+
"Keep": "pin"|"cache"|"discard"
81+
}
82+
}
83+
```
84+
85+
**Implementation steps:**
86+
87+
1. Implement everything but steps 3.a and 5.
88+
2. Implement step 3.a. We'd need this at a minimum before enabling this feature by default.
89+
3. Implement step 5. Unless nodes keep around a copy of the migration, this feature isn't going to be all that useful.
90+
91+
#### What does done look like?
92+
_What specific deliverables should completed to consider this project done?_
93+
94+
1. go-ipfs can download migrations over IPFS (bitswap) without having to contact a centralized server over HTTP.
95+
2. This feature has been tested by multiple labbers (possibly even tested by a subset of ipfs-desktop users).
96+
97+
#### What does success look like?
98+
_Success means impact. How will we know we did the right thing?_
99+
100+
Users can start a new version of go-ipfs that switches to a new repo version without downloading anything from a centralized (easy to block) server.
101+
102+
#### Counterpoints &amp; pre-mortem
103+
_Why might this project be lower impact than expected? How could this project fail to complete, or fail to be successful?_
104+
105+
1. Downloading migrations over IPFS could be too slow/unreliable. Falling back on a gateway should mitigate.
106+
2. The "temporary" node may be missing important parts of the configuration. For example, it may need alternative bootstrap nodes as our nodes may not be reachable. We'll need to think about this carefully and may need to "reach" into the old repo's config file a bit.
107+
108+
#### Alternatives
109+
_How might this project’s intent be realized in other ways (other than this project proposal)? What other potential solutions can address the same need?_
110+
111+
##### Domain Fronting
112+
113+
A very simple alternative is to simply use domain fronting. If we registered, e.g., ipfs-dist.com and made it an alias for dist.ipfs.io, it _might_ work in China.
114+
115+
However, we'd still be using HTTPS instead of IPFS to download migrations, which seems kind of silly.
116+
117+
##### Bundle the migrations
118+
119+
We could bundle all migrations with go-ipfs. But that would add 100s of megabytes to the go-ipfs distribution so we really don't want to do this.
120+
121+
#### Dependencies/prerequisites
122+
123+
We need to land https://github.com/ipfs/fs-repo-migrations/issues/98 first (in progress).
124+
125+
#### Future opportunities
126+
<!--What future projects/opportunities could this project enable?-->
127+
128+
This is a step towards self-bootstrapped, decentralized IPFS. The remaining parts are:
129+
130+
1. Updating go-ipfs itself over IPFS.
131+
2. Decentralized bootstrapping (reducing reliance on our central bootstrap nodes).
132+
133+
## Required resources
134+
135+
#### Effort estimate
136+
<!--T-shirt size rating of the size of the project. If the project might require external collaborators/teams, please note in the roles/skills section below).
137+
For a team of 3-5 people with the appropriate skills:
138+
- Small, 1-2 weeks
139+
- Medium, 3-5 weeks
140+
- Large, 6-10 weeks
141+
- XLarge, >10 weeks
142+
Describe any choices and uncertainty in this scope estimate. (E.g. Uncertainty in the scope until design work is complete, low uncertainty in execution thereafter.)
143+
-->
144+
145+
In terms of implementation, Small. But the testing/validation could take a variable amount of time.
146+
147+
#### Roles / skills needed
148+
149+
* Familiarity with go-ipfs, and the go-ipfs repo.
150+
* Ideally, familiarity with go-ipfs repo migrations.
151+
152+
The datasystems team would be the best fit.

0 commit comments

Comments
 (0)