Skip to content

Commit deb1330

Browse files
opensaucereropensaucerer
committed
feat: refactored ipfs-go-s3-ds to become compatible with the more recent migration to of go-ipfs to kubo
0 parents  commit deb1330

File tree

13 files changed

+3148
-0
lines changed

13 files changed

+3148
-0
lines changed

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
The MIT License (MIT)
2+
3+
Copyright (c) 2017 Whyrusleeping
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in
13+
all copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21+
THE SOFTWARE.

‎Makefile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Force Go Modules
2+
GO111MODULE = on
3+
4+
GOCC ?= go
5+
GOFLAGS ?=
6+
7+
# If set, override the install location for plugins
8+
IPFS_PATH ?= $(HOME)/.ipfs
9+
10+
# If set, override the IPFS version to build against. This _modifies_ the local
11+
# go.mod/go.sum files and permanently sets this version.
12+
IPFS_VERSION ?= $(lastword $(shell $(GOCC) list -m github.com/ipfs/go-ipfs))
13+
14+
# make reproducible
15+
ifneq ($(findstring /,$(IPFS_VERSION)),)
16+
# Locally built go-ipfs
17+
GOFLAGS += -asmflags=all=-trimpath="$(GOPATH)" -gcflags=all=-trimpath="$(GOPATH)"
18+
else
19+
# Remote version of go-ipfs (e.g. via `go get -trimpath` or official distribution)
20+
GOFLAGS += -trimpath
21+
endif
22+
23+
.PHONY: install build
24+
25+
go.mod: FORCE
26+
./set-target.sh $(IPFS_VERSION)
27+
28+
FORCE:
29+
30+
s3plugin.so: plugin/main/main.go go.mod
31+
CGO_ENABLED=1 $(GOCC) build $(GOFLAGS) -buildmode=plugin -o "$@" "$<"
32+
chmod +x "$@"
33+
34+
build: s3plugin.so
35+
@echo "Built against" $(IPFS_VERSION)
36+
37+
install: build
38+
install -Dm700 s3plugin.so "$(IPFS_PATH)/plugins/go-ds-s3.so"

‎README.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# S3 Datastore Implementation
2+
3+
This is an implementation of the datastore interface backed by amazon s3.
4+
5+
**NOTE:** Plugins only work on Linux and MacOS at the moment. You can track the progress of this issue here: https://github.com/golang/go/issues/19282
6+
7+
## Building and Installing
8+
9+
You must build the plugin with the *exact* version of go used to build the go-ipfs binary you will use it with. You can find the go version for go-ipfs builds from dist.ipfs.io in the build-info file, e.g. https://dist.ipfs.io/go-ipfs/v0.4.22/build-info or by running `ipfs version --all`.
10+
11+
In addition to needing the exact version of go, you need to build the correct version of this plugin.
12+
13+
* To build against a released version of go-ipfs, checkout the `release/v$VERSION` branch and build.
14+
* To build against a custom (local) build of go-ipfs, run `make IPFS_VERSION=/path/to/go-ipfs/source`.
15+
16+
You can then install it into your local IPFS repo by running `make install`.
17+
18+
## Bundling
19+
20+
As go plugins can be finicky to correctly compile and install, you may want to consider bundling this plugin and re-building go-ipfs. If you do it this way, you won't need to install the `.so` file in your local repo, i.e following the above Building and Installing section, and you won't need to worry about getting all the versions to match up.
21+
22+
```bash
23+
# We use go modules for everything.
24+
> export GO111MODULE=on
25+
26+
# Clone go-ipfs.
27+
> git clone https://github.com/ipfs/go-ipfs
28+
> cd go-ipfs
29+
30+
# Pull in the datastore plugin (you can specify a version other than latest if you'd like).
31+
> go get github.com/ipfs/go-ds-s3/plugin@latest
32+
33+
# Add the plugin to the preload list.
34+
> echo -en "\ns3ds github.com/ipfs/go-ds-s3/plugin 0" >> plugin/loader/preload_list
35+
36+
# ( this first pass will fail ) Try to build go-ipfs with the plugin
37+
> make build
38+
39+
# Update the deptree
40+
> go mod tidy
41+
42+
# Now rebuild go-ipfs with the plugin
43+
> make build
44+
45+
# (Optionally) install go-ipfs
46+
> make install
47+
```
48+
49+
## Detailed Installation
50+
51+
For a brand new ipfs instance (no data stored yet):
52+
53+
1. Copy s3plugin.so $IPFS_DIR/plugins/go-ds-s3.so (or run `make install` if you are installing locally).
54+
2. Run `ipfs init`.
55+
3. Edit $IPFS_DIR/config to include s3 details (see Configuration below).
56+
4. Overwrite `$IPFS_DIR/datastore_spec` as specified below (*Don't do this on an instance with existing data - it will be lost*).
57+
58+
### Configuration
59+
60+
The config file should include the following:
61+
```json
62+
{
63+
"Datastore": {
64+
...
65+
66+
"Spec": {
67+
"mounts": [
68+
{
69+
"child": {
70+
"type": "s3ds",
71+
"region": "us-east-1",
72+
"bucket": "$bucketname",
73+
"rootDirectory": "$bucketsubdirectory",
74+
"accessKey": "",
75+
"secretKey": ""
76+
},
77+
"mountpoint": "/blocks",
78+
"prefix": "s3.datastore",
79+
"type": "measure"
80+
},
81+
```
82+
83+
If the access and secret key are blank they will be loaded from the usual ~/.aws/.
84+
If you are on another S3 compatible provider, e.g. Linode, then your config should be:
85+
86+
```json
87+
{
88+
"Datastore": {
89+
...
90+
91+
"Spec": {
92+
"mounts": [
93+
{
94+
"child": {
95+
"type": "s3ds",
96+
"region": "us-east-1",
97+
"bucket": "$bucketname",
98+
"rootDirectory": "$bucketsubdirectory",
99+
"regionEndpoint": "us-east-1.linodeobjects.com",
100+
"accessKey": "",
101+
"secretKey": ""
102+
},
103+
"mountpoint": "/blocks",
104+
"prefix": "s3.datastore",
105+
"type": "measure"
106+
},
107+
```
108+
109+
If you are configuring a brand new ipfs instance without any data, you can overwrite the datastore_spec file with:
110+
111+
```
112+
{"mounts":[{"bucket":"$bucketname","mountpoint":"/blocks","region":"us-east-1","rootDirectory":"$bucketsubdirectory"},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}
113+
```
114+
115+
Otherwise, you need to do a datastore migration.
116+
117+
## Contribute
118+
119+
Feel free to join in. All welcome. Open an [issue](https://github.com/ipfs/go-ipfs-example-plugin/issues)!
120+
121+
This repository falls under the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md).
122+
123+
### Want to hack on IPFS?
124+
125+
[![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md)
126+
127+
## License
128+
129+
MIT

‎docker-compose.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
version: "3"
2+
services:
3+
minio:
4+
image: minio/minio:edge@sha256:373cc16a29662f3f616a8de7facff0ef085257296256c59d64b158d9a59d2bd9
5+
environment:
6+
- MINIO_REGION_NAME=local
7+
- MINIO_ROOT_USER=test
8+
- MINIO_ROOT_PASSWORD=testdslocal
9+
ports:
10+
- 9000:9000
11+
command:
12+
- server
13+
- /data

0 commit comments

Comments
 (0)