How to load embeddings of 500 pages document

[deepak-hl]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

loader = PDFPlumberLoader(file_path)
            document = loader.load()
            # text_splitter = TokenTextSplitter(chunk_size=2000, chunk_overlap=200)
            text_splitter = RecursiveCharacterTextSplitter(chunk_size=6000, chunk_overlap=100)
            texts = text_splitter.split_documents(documents=document)
            embedding = OpenAIEmbeddings()
 
            PGVector.from_texts(
                    embedding=embedding,
                    texts=texts,
                    collection_name=persist_directory,
                    connection_string=CONNECTION_STRING,
                )

Description

I am trying to upload 500 pages pdf, but during running above code(embedding process), my laptop stucks and hanged,
Can't we do this process asyncrously or what modification need to do in my code to do so?

System Info

langchain
linux

[dosubot[bot]]

To load embeddings for a 500-page PDF document without causing your laptop to become unresponsive, you can process the PDF in smaller batches. This approach will help manage memory usage and prevent your system from becoming overwhelmed. Here is an updated version of your code that processes the PDF in batches:

import os
from pathlib import Path
import pdfplumber
from langchain.vectorstores import PGVector
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter

def process_pdf_in_batches(file_path, batch_size=50):
    with pdfplumber.open(file_path) as pdf:
        n_pages = len(pdf.pages)
        for start_page in range(0, n_pages, batch_size):
            end_page = min(start_page + batch_size, n_pages)
            yield [pdf.pages[i].extract_text() for i in range(start_page, end_page)]

# Load PDF
file_path = "path/to/your/500_pages_document.pdf"

# Load embedding function
embedding = OpenAIEmbeddings()

# Create text splitter
text_splitter = RecursiveCharacterTextSplitter(chunk_size=6000, chunk_overlap=100)

# Process PDF in batches
for batch_texts in process_pdf_in_batches(file_path):
    documents = [{"page_content": text} for text in batch_texts]
    texts = text_splitter.split_documents(documents=documents)
    
    # Add embeddings
    PGVector.from_texts(
        embedding=embedding,
        texts=texts,
        collection_name="your_collection_name",
        connection_string="your_connection_string",
    )

print("done")

This code processes the PDF in batches of 50 pages at a time, which should help prevent your laptop from becoming unresponsive. Adjust the batch_size as needed based on your system's capabilities.

Additionally, LangChain has built-in asynchronous methods or utilities for processing large documents. For instance, the BaseDocumentCompressor class includes an asynchronous method acompress_documents, and the ReduceDocumentsChain class provides the acombine_docs method for asynchronous document processing ^[1][2]. You can explore these methods to further optimize your code for asynchronous processing.

To continue talking to Dosu, mention @dosu.