Migration Tooling - 20th March - Storacha Migration Tool - PLDG 2

[HarshS1611]

Migration Tooling - 20th March - Storacha Migration Tool - PLDG 2

Contributors

@HarshS1611 @leny62

Accomplished Tasks

• Created a progress bar for real-time process.
• Successfully tested uploading a large file (2GB).
• Added matrix such as time remaining, number of files, speed etc.

Update

• Added a real time progress bar to show the progress of fetching the file from S3 and uploading it to storacha space.
• Added more matrixes to improve the UX:
  • Number of files to be uploaded.
  • Uploading speed.
  • Size of the total file/folder to be uploaded.
  • Estimated time remaining to be migrated.
• Successfully migrated a large file (2GB) from S3 to storacha.

Demo

IPFS LINK

Questions

• Any resources for the fail-safe issue (resuming the upload after failing to upload) would be helpful.
• While using the new email first time for using storacha services, the user has to verify the email from web3.storage. But it's not showing on the console. Can it be shown to the user?

Next Tasks

• Testing edge cases if the uploading fails and if it resumes or starts uploading the complete file again.
• Defining types to all the migrator functions.
• While changing the email address for storacha, it is still using the previous email's space. So, we will try to fix this issue.

[leny62]

• We need to figure out a way for storacha upload progress as well.

[jamiechicago312]

Heyyy

https://github.com/storacha/w3up/blob/2ddb4042fee59e6b667057f6e82c711fa9e318ee/packages/upload-client/src/types.ts#L155

This is where you can check out the types available for onprogress status

[jamiechicago312]

This is more in depth about progress

Upload Progress Reporting

onUploadProgress callback can be passed as an option to uploadFile, uploadDirectory or uploadCAR.

https://github.com/storacha/w3up/blob/2ddb4042fee59e6b667057f6e82c711fa9e318ee/packages/upload-client/src/types.ts#L317

Alternatively onShardStored callback is called after a CAR shard has been stored and can be used to communicate progress for larger uploads.

https://github.com/storacha/w3up/blob/2ddb4042fee59e6b667057f6e82c711fa9e318ee/packages/upload-client/src/types.ts#L393C3-L393C16

Shards are max 128MB (although this can also be configured).

Advanced

You can pass a fetchWithUploadProgress option for more fine grained progress reporting in browser environments. It uses XHR for this.

Import and use like so (note: this is browser only).

import { fetchWithUploadProgress } from '@web3-storage/upload-client/fetch-with-upload-progress'

client.uploadFile(new Blob([]), { fetchWithUploadProgress })

[HarshS1611]

Thanks for the help. We'll make sure to update you soon.

[jamiechicago312]

Just took some time to read & review your update properly. WOW 🤩 Thank you for taking the feedback from last week and implementing this in our demo. Love being able to see the progress. Excited to see what you come up with for next week. Thank you for your diligent work @HarshS1611 and @leny62 🙌