Storacha RAG Agent - March 24,2025 - PLDG2

[adielliot37]

AI Agents / RAG - March 24, 2025 - Storacha RAG Agent - PLDG 2

Contributors

@adielliot37

Accomplished Tasks

•	Integrated Storacha with a Retrieval-Augmented Generation (RAG) pipeline.
•	Implemented chunking and IPFS-based storage of knowledge fragments.
•	Enabled support for uploading and querying from:
•	Plain text
•	PDFs
•	URLs
•	Successful end-to-end flow: Upload → Chunk → Embed → Store on Storacha → Query with Mistral

Update

•	Built a decentralized RAG agent where knowledge fragments are stored using Storacha on IPFS, replacing traditional centralized vector databases.
•	Chunks of data (from text, PDFs, and URLs) are stored as JSON objects on IPFS and indexed with embeddings for semantic search.
•	Fetching relevant chunks during querying works reliably across multiple IPFS gateways.
•	What went well:
•	Integration with Storacha was smooth.
•	Embedding, chunking, and storage pipeline is fully functional.
•	What was difficult:
•	Fetching from IPFS occasionally times out or fails on some gateways.
•	Minor issues parsing malformed PDFs or inconsistent webpage structures.

Demo Videos

Questions

•	How can I improve the uploading and fetching speed from Storacha/IPFS?
•	Is there a way to prioritize gateway availability or caching strategies for faster retrieval?
•	Would it be possible to integrate on-chain metadata verification for knowledge chunks?

Next Tasks

•	Add UCAN support so multiple agents can delegate and access a common decentralized knowledge base.
•	Explore compression or pre-processing to optimize chunk size before uploading.
•	Implement multi-gateway fallback with retries for more reliable fetching.
•	Add basic auth and namespacing to organize chunks per user/project.
•	Explore automated sync/backfill agent that continuously listens for new knowledge fragments.

[jamiechicago312]

WOW this is awesome! I love how you were able to upload data and your agent was able to use that to inform their answers.