multiprocessing for VectorStoreIndexCreator

[IamExperimenting]

Hi Team,

I'm using LangChain to for my use-case, while creating VectorStoreIndexCreator is takes longer time to complete it. I believe it process pdf document squentially.

from langchain.indexes import VectorStoreIndexCreator
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.text_splitter import CharacterTextSplitter

index = VectorStoreIndexCreator(
                      embeddings = HuggingFaceEmbeddings(),
                      text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)).from_loaders(loaders)

Is there a better approach to improve its time? like multiprocessing or async way to complete this work faster and utilise the cpu efficiently.

Currently, I have 2000 pdf files, each file has around 30 pages.

It would helpful if you provide some sample/example.

[oddrationale]

This blog post shows a way to use Ray and AWS to parallelize the processing of embeddings:
Turbocharge LangChain NOW: guide to 20x faster embedding

[IamExperimenting]

Hi @oddrationale , the blog which you have shared with me, it uses "anyscale- ray", However, I'm using SageMaker, can I use multi-gpu instance to do this? because i'm not familiar in working with clusters. Could you please help me here?

[oddrationale]

Unfortunately, I have not implemented this myself and not able to give you more guidance that what I found in the blog post and what you can find by searching the web. Sorry!

[IamExperimenting]

I tried with multi-gpu instance, I don't know whether it uses GPU, but I presume it uses CPU and does parallel processing.