Add integration for pymupdf4llm as one of the PDF loaders

[zymbuzz]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

I would like to suggest adding PyMuPDF4LLM as another PDF loader for the langchain.

Motivation

The integration would allow the package's capabilities to be used to better parse multiple formats of text data that adhere to intricacies for better RAG.

Proposal (If applicable)

No response

[RafikYH]

Hello, i managed to use it as a loader by changing it's output, check code down below (i'm sure it can be better written) :

from pymupdf4llm import to_markdown

def convert_pdfs_to_markdown(path: str):
    # List to hold the markdown contents of all PDFs
    markdown_contents = []
    md_langchain = []

    if os.path.isfile(path) and path.endswith(".pdf"):
        # Single PDF file case
        file_path = path
        try:
            # Convert the PDF to markdown
            md_text = to_markdown(file_path, page_chunks=True)
            markdown_contents.append(md_text)
        except Exception as e:
            print(f"Failed to convert {file_path}: {e}")
    else:
        # Directory case
        for root, _, files in os.walk(path):
            for file in files:
                if file.endswith(".pdf"):
                    file_path = os.path.join(root, file)
                    print(file_path)
                    try:
                        # Convert the PDF to markdown
                        md_text = to_markdown(file_path, page_chunks=True)
                        markdown_contents.append(md_text)
                    except Exception as e:
                        print(f"Failed to convert {file_path}: {e}")

    # Convert the markdown contents to LangChain format
    for file in markdown_contents:
        langchain_output = llamaindex_to_langchain(file)
        md_langchain.extend(langchain_output)

    return md_langchain


class Document:
    def __init__(self, page_content: str, metadata: Dict[str, Any]):
        self.page_content = page_content
        self.metadata = metadata

    def __repr__(self):
        return f"Document(page_content={repr(self.page_content)}, metadata={self.metadata})"


def llamaindex_to_langchain(llamaindex_output: List[Dict[str, Any]]) -> List[Document]:
    langchain_output = []

    for item in llamaindex_output:
        # Extract the text content and metadata from the LlamaIndex dictionary
        text = item["text"]
        metadata = item["metadata"]

        # Transform the metadata to match the LangChain structure
        langchain_metadata = {
            "source": metadata["file_path"],
            "file_path": metadata["file_path"],
            "page": metadata["page"],
            "total_pages": metadata["page_count"],
            "format": metadata["format"],
            "title": metadata["title"],
            "author": metadata["author"],
            "subject": metadata["subject"],
            "keywords": metadata["keywords"],
            "creator": metadata["creator"],
            "producer": metadata["producer"],
            "creationDate": metadata["creationDate"],
            "modDate": metadata["modDate"],
            "trapped": metadata["trapped"],
        }

        # Create a new Document object for the LangChain output
        document = Document(page_content=text, metadata=langchain_metadata)
        langchain_output.append(document)

    return langchain_output

Now that you have the function ready, the way i use it is :

def generate_data_store(data_path, st_chroma_path):
    documents = load_documents(data_path)
    chunks = split_text(documents)
    save_to_st_chroma(chunks, st_chroma_path)


def load_documents(data_path):
    documents = convert_pdfs_to_markdown(data_path)
    print("LOADER : \tLoaded file successfully.")
    return documents

[0xMatthieu]

Thank you @RafikYH it works well !

[rroy0412]

split_text--> what splitter did you use for these markdown files?

[0xMatthieu]

@rroy0412

this simple snippet should work

from langchain.text_splitter import RecursiveCharacterTextSplitter
from pymupdf4llm import to_markdown
loader = to_markdown(filepath, page_chunks=True, write_images=False)
document = llamaindex_to_langchain(loader)
text_splitter = RecursiveCharacterTextSplitter (chunk_size=500, chunk_overlap=200)
texts = text_splitter.split_documents(document)

[lakinduboteju]

I just published the initial version of a PyMuPDF4LLM integration package to PyPi.
https://pypi.org/project/langchain-pymupdf4llm

Source code is at: https://github.com/lakinduboteju/langchain-pymupdf4llm

Related to: #29848

[lakinduboteju]

https://python.langchain.com/docs/integrations/document_loaders/pymupdf4llm