You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The helper packs the file into a CARv1 with `npx ipfs-car` (defaults
60
+
match the unixfs-v1-2025 / IPIP-0499 profile so CIDs are reproducible),
61
+
uploads the CAR to your [Filebase] IPFS bucket via Filebase's S3-compatible
62
+
REST API, verifies the CID Filebase reports back matches what was computed
63
+
locally, and replaces the original file with `MyTest.png.cid` containing
64
+
that CID. The CID and source-tree path are also recorded in
65
+
`Testing/Data/content-links.manifest`. A local IPFS daemon is **not**
66
+
required.
67
+
68
+
First-time CLI users must complete the one-time pixi + Filebase setup
69
+
documented in
70
+
[`Utilities/Maintenance/ExternalDataUpload/README.md`] before the helper
71
+
will succeed. Contributors who prefer not to run any local tooling can
72
+
instead use the [content-link-upload] web app, which returns a `.cid`
73
+
file directly.
47
74
48
75
For more details, see the description and procedures in [Upload Binary Data].
49
76
@@ -142,5 +169,8 @@ the [InterPlanetary File System (IPFS)].
142
169
[CMake ExternalData: Using Large Files with Distributed Version Control]: https://blog.kitware.com/cmake-externaldata-using-large-files-with-distributed-version-control/
55
61
56
-
*Testing data workflow. Testing or example data is uploaded to IPFS via the content-link-upload.itk.org web app.
57
-
This pins the data on multiple servers across the globe.
58
-
At release time, the data is also pinned on multiple servers in the USA and France and community pinners.
59
-
At release time, the data is also stored in the DataLad Git repository, served on an Apache HTTP server, and the GitHub Pages CDN.
60
-
At test time an ITK build can pull the data from a local cache, archive tarball, the Apache HTTP server, GitHub Pages CDN, or multiple IPFS HTTP gateways.*
61
-
62
-
See also our [Data](data.md) guide for more information. If you just
63
-
want to browse and download the ITK testing images, see the
64
-
[ITKData DataLad repository].
62
+
*Testing data workflow. New content is added with the
63
+
`Utilities/Maintenance/ExternalDataUpload/upload.py` helper, which packs
64
+
the file into a CAR with `npx ipfs-car` (defaults match the
65
+
unixfs-v1-2025 / IPIP-0499 profile so CIDs are reproducible) and uploads
66
+
the CAR to a [Filebase] IPFS bucket via boto3 against Filebase's
67
+
S3-compatible API. The CID Filebase reports back from `head_object` is
68
+
verified against the locally computed CID, written as a `.cid` content
69
+
link in the ITK source tree, and recorded in
70
+
`Testing/Data/content-links.manifest`. Files ≤ 50 MB can additionally be
71
+
mirrored into [ITKTestingData] for GitHub Pages CDN delivery. At test
72
+
time an ITK build can fetch the data from a local cache, archive tarball,
73
+
the Apache HTTP server, the GitHub Pages mirror, or any of several public
74
+
IPFS HTTP gateways.*
75
+
76
+
See also our [Data](data.md) guide for more information.
65
77
66
78
Adding images as input to ITK sources
67
79
-------------------------------------
@@ -89,107 +101,136 @@ need to be followed:
89
101
Upload new testing data
90
102
-----------------------
91
103
92
-
### Prerequisites
104
+
### One-time setup
93
105
94
-
[web3.storage] is a decentralized IPFS storage
95
-
provider where any ITK community member can upload binary data files.
96
-
There are two primary methods available to upload data files:
106
+
The upload workflow needs:
97
107
98
-
A. The CMake ExternalData Web3 upload browser interface.
99
-
B. The <spanclass="title-ref">w3</span> command line executable that
100
-
comes with the [@web3-storage/w3cli] Node.js NPM package.
108
+
- The `external-data-upload` pixi environment installed
109
+
(`pixi install -e external-data-upload`). It provides Python 3, [boto3],
110
+
and Node.js (which makes `npx ipfs-car` available without a separate
111
+
global install).
112
+
- A [Filebase] IPFS bucket and an S3 access key for that bucket. Filebase's
113
+
free tier is sufficient — the upload uses the S3 import-as-CAR path,
114
+
not the legacy IPFS Pinning Service API.
115
+
- The credentials exported as environment variables before running the
116
+
helper:
101
117
102
-
Once files have been uploaded, they will be publicly
103
-
available and accessible since data is content addressed on the IPFS
104
-
peer-to-peer network.
118
+
```bash
119
+
export FILEBASE_ACCESS_KEY=...
120
+
export FILEBASE_SECRET_KEY=...
121
+
export FILEBASE_BUCKET=itk-data
122
+
```
105
123
106
-
In addition to these two methods, documented in detail below, another
107
-
possibility includes pinning the data on IPFS with [other pinning services]
108
-
and creating the content link file manually. The content link file is simply a
109
-
plan text file with a `.cid` extension whose contents are the CID file.
110
-
However, the documented two methods are recommended due to their simplicity
111
-
and in order to keep CID values consistent.
124
+
The full step-by-step setup is documented in
125
+
[`Utilities/Maintenance/ExternalDataUpload/README.md`]. Complete that
126
+
one-time setup before proceeding.
112
127
113
-
At release time, the release manager uploads and archives repository data
114
-
references in other storage locations for additional redundancy.
128
+
[boto3]: https://boto3.amazonaws.com/
115
129
116
-
### Option A) Upload Via the Web Interface
130
+
### Upload a file
117
131
118
-
Use the [Content Link Upload]
119
-
tool ([Alt Link]) to
120
-
upload your data to the [IPFS] and download the
121
-
corresponding CMake content link file.
132
+
From the ITK source tree, run the upload helper with the path to the file
133
+
you want to upload:
122
134
123
-
](./content-link-upload.png)
0 commit comments