Memory usage sitemap loader & vectorstore

[eastuger]

Hi I'm running in to memory issues using sitemap loaders and Pinecone vector storage.

My server is frequently crashing due to memory limit which is currently 16GB of ram. I don't really think the issue is retrieving all the pages the memory usage is pretty stable. But when I start indexing the server can crash.

Increasing the memory limit to 32GB was a quick and dirty solution. But is there any other way I can solve this using less memory.

Loader code

    try:
        sitemap = LoaderProvider(self.provider)
        sitemap = sitemap.get_loader()
        sitemap_loader = sitemap(
            f"{self.document.name}/sitemap.xml",
            parsing_function=remove_nav_and_header_elements,
        )

        self.documents = sitemap_loader.load_and_split(
            RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
        )

Vector store code

    for name in pinecone.list_indexes():
        index, stats = self.get_stats(name)

        if self.namespace in stats.namespaces:
            self.index_name = name
            index.delete(delete_all=True, namespace=self.namespace)
            break
        else:
            continue

    try:
        Pinecone.from_documents(
            documents,
            OpenAIEmbeddings(),
            namespace=self.namespace,
            index_name=self.index_name,
        )

[dosubot[bot]]

🤖

Hi,

Thanks for reaching out. It seems like you're dealing with memory issues when using sitemap loaders and Pinecone vector storage. Here are some strategies you can consider to optimize memory usage:

1. Batch Processing: Instead of loading all the data at once, you can load and process your data in smaller chunks. This can be done using the batch_size parameter in the add_texts method. This will reduce the amount of memory used at any one time. Here's an example:

# Assume texts is a large list of texts to be added
batch_size = 1000  # Adjust this value based on your memory capacity
for i in range(0, len(texts), batch_size):
    batch_texts = texts[i:i+batch_size]
    vectorstore.add_texts(batch_texts)

2. Delete Unnecessary Data: If there are vectors in your Pinecone vector store that are no longer needed, you can delete them using the delete method. This will free up memory. Here's how you can delete vectors:

# Assume ids_to_delete is a list of ids of vectors to be deleted
vectorstore.delete(ids=ids_to_delete)

3. Optimize Embeddings: If you're using large embeddings, consider using a dimensionality reduction technique to reduce the size of your embeddings. This will reduce the amount of memory required to store and process these embeddings.

4. Sitemap Loader Blocksize and Blocknum: In the context shared, the SitemapLoader in LangChain has a blocksize parameter and a blocknum parameter. The blocksize parameter can be used to control the number of URLs loaded from the sitemap at a time. By reducing the blocksize, you can reduce the amount of memory used. The blocknum parameter can be used to specify which block of URLs to load, allowing you to load different blocks at different times, further reducing memory usage. Here's how you can adjust the blocksize and blocknum when using the SitemapLoader:

# Assume you want to load URLs from a sitemap in blocks of 1 and load the first block
loader = SitemapLoader("https://langchain.readthedocs.io/sitemap.xml", blocksize=1, blocknum=0)
documents = loader.load()

Remember to adjust the batch_size parameter, the list of ids to delete, and the blocksize and blocknum parameters according to your specific needs and memory capacity.

I hope this helps! Let me know if you have any other questions.

Best,
Dosu

Sources

• libs/langchain/langchain/vectorstores/pinecone.py
• libs/langchain/tests/integration_tests/document_loaders/test_sitemap.py
• libs/langchain/langchain/document_loaders/sitemap.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.