Where is original text data stored?

[etjm1993]

Hi there!

When I upload a document to Flowise and then upsert it to Pinecone, where is the original text stored and then how is it accessed when I ask a question to the chat?

I want to create a process of automatically creating embeddings and upserting them to pinecone so that I do not have to do it manually but I am struggling to find the answer to my question above.

Any help or direction would be highly appreciated!

Thank you

[HenryHengZJ]

The original text is converted into a base64 encoded string and stored as part of your chatflow data. When you ask a question in the chat when using any upsert node, it will automatically create embeddings and upsert data

[jeffrey-devhkg]

Is this the same with pdf file upload?

[chungyau97]

Hi @jeffrey-devhkg, yes it's the base64 of the pdf file upload.

[jeffrey-devhkg]

Thanks!

[jeffrey-devhkg]

Is the encoded base64 of the pdf stored in database.sqlite?

[HenryHengZJ]

yes

[etjm1993]

Got it, I am looking for a way to automate the process of creating embeddings and upserting the data whilst retaining the original text. Is there a way to do this with flowise? Thanks!

…

On Fri, 1 Sep 2023 at 10:15, Henry Heng ***@***.***> wrote: The original text is converted into a base64 encoded string and stored as part of your chatflow data. When you ask a question in the chat when using any upsert node, it will automatically create embeddings and upsert data — Reply to this email directly, view it on GitHub <#859 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A63WJ6JIB6HH3UMHS5GPYCDXYGRSHANCNFSM6AAAAAA4GUNIVQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[HenryHengZJ]

This is not possible yet on Flowise, definitely hope to build this feature in future

[etjm1993]

No it was with json documents. The way I solved it was to index them into vectara and then they are ready to use via flowise without any embeddings or storing of the text elsewhere!

…

On Thu, 21 Sep 2023 at 10:24, Jeffrey Yim ***@***.***> wrote: Is this the same with pdf file upload? — Reply to this email directly, view it on GitHub <#859 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A63WJ6J3347ACWYVI63XIGLX3QBVLANCNFSM6AAAAAA4GUNIVQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[chungyau97]

Hi @etjm1993, Vectara itself have built-in embedding on their platform.