Migration Tooling - 03rd April - Storacha Migration Tool - PLDG 2

[leny62]

NextJS UI Migration Tool: Browser Integration for Storacha

Contributors

• @HarshS1611
• @leny62

Accomplished Tasks

• ✅ Created browser-compatible wrapper for the Storacha Migration Tool
• ✅ Implemented server-side API endpoints for S3 and Storacha operations
• ✅ Developed responsive UI with real-time progress visualization
• ✅ Separated S3 download and Storacha upload flows
• ✅ Added real-time progress tracking with speed and time estimates
• ✅ Implemented file handling with proper multipart uploads
• ✅ Created sharded upload visualization for large files

Update

We've successfully built a browser-compatible UI for the Storacha Migration Tool:

S3 Integration

• Created a dedicated API endpoint that directly uses AWS SDK for S3 operations
• Implemented file metadata retrieval to properly handle downloads
• Added progress simulation on the client side for better UX during server-side operations

Storacha Integration

• Built a multipart file upload system using FormData and XHR for progress tracking
• Implemented proper file handling with the StorachaMigrator library

UI Components

• Created reusable ProgressBar component with customizable styling
• Implemented StorachaUploader with real-time progress, speed, and time estimates
• Developed S3Downloader with comprehensive error handling and status display
• Added utility functions for formatting bytes, time, and speed

Browser Compatibility

• Created BrowserMigrator wrapper to handle Node.js dependencies in browser
• Implemented mock modules for fs, path, and other Node.js built-ins
• Added fallback simulation mode when library can't be loaded
• Enhanced webpack configuration to properly handle module aliases

Demo

mig-tool-ui.mp4

(Full demo to be shared in comment)

Questions

• Should we implement a file browser for S3 navigation instead of requiring exact object keys?
• Do we still need to authenticate to storacha with email or there is a way to use storacha api keys
• Should we add support for cancelling in-progress migrations or resuming interrupted uploads?
• Are there specific customizations needed for different types of files or storage patterns?
• How should we display a user's current active uploads across multiple sessions?
• What's the best way to visualize a user's space list and allow easy switching between spaces?
• Should we implement a space usage dashboard with storage metrics and quotas?

Next Tasks

• 🔄 Add proper authentication flow for Storacha login
• 🔄 Implement directory upload support with tree visualization
• 🔄 Add batch processing for multiple files with queue management
• 🔄 Create a dashboard for monitoring active and completed migrations
• 🔄 Implement proper error recovery for failed uploads
• 🔄 Optimize upload progress tracking with fine-grained chunk reporting
• 🔄 Implement analytics to track upload performance and success rates
• 🔄 Add space management interface with detailed usage statistics

Architecture

This implementation uses a hybrid client-server approach:

• Client components handle UI and progress visualization
• Server API endpoints execute actual S3 and Storacha operations
• Browser-compatible wrappers handle Node.js dependencies

This approach provides the best user experience while maintaining the security and functionality of the core migration library.

Future Optimization Plans

We plan to significantly enhance the upload experience in upcoming iterations:

1. Fine-grained Progress Tracking: Implement chunk-based progress tracking to provide more accurate progress indications, especially for large files.

2. Analytics Dashboard: Add a comprehensive analytics system to track:

   • Upload success rates and failure patterns
   • Average upload speeds and completion times
   • Storage usage patterns across spaces
   • File type distribution and access patterns

3. Concurrent Transfer Management: Develop an intelligent queue system to optimize bandwidth usage and prioritize critical transfers.

[jamiechicago312]

Hey all 👋

Questions from last post also:

Are there any specific metrics we should add to the progress display beyond what we currently have?

What are the recommended practices for handling large directory migrations with thousands of files?
Possibly yes, there are limits within call stack js. This thread in our discord server talked about uploading 20k files at one time: https://discord.com/channels/1247475892435816553/1355220191793123679 [invite to discord for those who a reading & can't access the link]

Should we implement any throttling mechanisms to prevent overwhelming the Filecoin nodes?
You should be OK; Storacha will handle the upload to filecoin.

This week's questions:

Should we implement a file browser for S3 navigation instead of requiring exact object keys?
Mmm, i'd focus on creating a tool the does the entire mirgation versus select files. I believe selecting certain files is more of a feature.

Do we still need to authenticate to storacha with email or there is a way to use storacha api keys
You can use UCANs to create delegations to upload to a space from the owner of the space. We did a blog post write up on it this week. But we don't have API keys because they are centralized and prone to getting leaked.

Should we add support for cancelling in-progress migrations or resuming interrupted uploads?
It would be good if it could resume after being interrupted. I think instead of cancelling pausing might be better. To cancel I think implies to then go and delete what you already uploaded too.

Are there specific customizations needed for different types of files or storage patterns?
Not sure, i'd ask in Discord.

How should we display a user's current active uploads across multiple sessions?
Up to you, I would focus on making the migration tool work before adding this feature.

What's the best way to visualize a user's space list and allow easy switching between spaces?
Probably a file directory like we have in the console: https://console.storacha.network

Should we implement a space usage dashboard with storage metrics and quotas?
Up to you, I would focus on making the migration tool work before adding this feature.

[HarshS1611]

Thanks @jamiechicago312 for all the help and feedback.