Summary + Large Context RAG - Storing Text Chunks in Flowise

[BadrLaajali]

Hi,
I've been working on a project where I aim to implement a "Summary + Large Context RAG" strategy. The idea is to:

Chunking Large Texts: Break a large document into sizable chunks (e.g., around 1300 words each). Each of these chunks is assigned a unique identifier.
Storing Chunks: Save these chunks, along with their unique identifiers, into an external storage, like an Excel file.
Creating Summaries: For each chunk, generate a concise summary using a language model.
Indexing Summaries: Store these summaries in a vector database (like Pinecone) with the unique identifier pointing back to the original large chunk.
Retrieval: At query time, when a summary matches a user's question, use the unique identifier to retrieve the original large chunk from the Excel file for further processing.
However, I've encountered a challenge with Flowise. While I can chunk large texts and generate summaries, I'm struggling to directly save the chunks with their unique identifiers into an Excel file using Flowise. I've tried setting up a custom tool that triggers automation on Make, but I'm blocked without the ability to directly extract the chunks of text from Flowise.

Is there a direct method or tool within Flowise that allows for this kind of operation? Or perhaps there's a workaround that I might not be aware of? Any guidance or feature additions in this direction would be immensely helpful.

Thank you for your time and assistance!

[BigFoxMedia]

Great initiative though I can't understand the choice to use Excel. I would highly recommend reassessing this choice in favor of mongo or sql or any other db based solution just not excel. It is not a proper data store solution and not a scalable one either. I don't expect any serious answer from the community to find an answer to this task using Excel as the back bone data store.

Please update oncyour progress and in particular I would love to know the accuracy of the resulting llm response , if you don't mind sharing.

Good luck!

[BadrLaajali]

Thanks for your feedback. I agree that Excel is not a good choice for a production solution. I am currently using it for testing purposes only. I am developing a final solution that will use Airtable or MongoDB.
My question is more about the fact that I can't retrieve a variable in Flowise that stores the output of a chunk.