BaseDocumentTransformer must be a Runnable

[pprados]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

BaseDocumentTransformer must be synchrone, asynchrone, a pipeline of transformations, etc.
It's the specification of Runnable.

Why ̀BaseDocumentTransformeris not compatible with LCELRunnable`?

Motivation

It's simplest to write transform1 | transform2 or more specific (transform1 | transform2 ) & transform3 to generate multiple versions from the same source.

A,B => transform2(transform1(A), transform3(A), transform2(transform1(B), transform3(B)

• transform1 may be to upper
• transform2 may be 'split token'
• transform3 may be 'summarize`

etc.

Proposal (If applicable)

No response

[eyurtsev]

To incorporate into LangChain, we'd want a clean design that handles a few scenarios. This functionality is unlikely to produce a significant benefit for most users, so we haven't dedicated any time to speccing it out.

Main complications:

1. Transformations may be compute or IO bound -- this is the main issue that needs to be addressed before composition using LCEL is useful
2. Runnable interface requires an input, but BaseDocumentLoader.lazy_load() is parameterless.

Feel free to propose an RFC -- for a good RFC, code should be as minimal as possible, with a simple and short explanation of the contents.

[pprados]

I've already proposed an implementation, but the PR was closed without explanation.

Can I have your support in resynchronising the proposal with version 0.2?

[eyurtsev]

Likely the PR wasn't closed by the LangChain team. There was an issue on the github side that closed all open PRs against langchain.

I don't know if the original PR had any code, but I don't think the PR description explains how to handle the two main problems in general way.

I don't know if there's much benefit in just introducing a lazy transform approach since the user can do it as is right now? Is there much benefit from using LCEL vs. just a for loop?

[pprados]

@eyurtsev

Transforming lazily using a loop does not allow us to remain lazy. Indeed, the loop must iterate over all documents to produce a transformation each time. The result is an array, not a lazy generator. The current implementations of TextSplitter are not capable of working via an iterator. They only know how to produce a list of documents in memory.

When transforming a large number of documents, this consumes memory and breaks the ability to use the lazy approach when using index(). It is necessary to start with a lazy loader, then perform the transformations to get a list of transformed documents, before passing it to the index. As a reminder, it is possible to use index() with a lazy_load().

Ideally, index() should be able to accept a lazy loader and work with a lazy transformation. This would prevent unnecessary memory consumption and be consistent with the batch_size parameter of index()̀. The batch can be lazy!

Currently, the only solution for importing a very large list of documents is to saturate the memory.

What is the second need?
The implementation of ParentDocumentRetriever does not seem appropriately positioned. Indeed, it is a retriever that offers a method add_documents(). This is quite strange for a retriever, and it is incompatible with index().

I propose an alternative, RAGVectorstore (see here), which provides the same service as ParentDocumentRetriever, but is consistent with index().

Whether it is for ParentDocumentRetriever or RAGVectorstore, it is necessary to set the parent_splitter and child_splitter attributes
The implementation of ParentDocumentRetriever only accepts textsplitter, which is a limitation. It is not possible, for example, to generate a summary of a chunk, or to ask questions for each chunk (the call MUST be synchrone !).

How can we initialize these splitters? I propose a DocumentTransformerPipeline() class to allow for a pipeline transformation.

wiki_splitter = RecursiveCharacterTextSplitter(
    separators=[
        "={1,6} .* ={1,6}",  # See https://en.wikipedia.org/wiki/Help:Wikitext
        "\n----+\n",            
        "\n\n",
        "\n",
    ],
    chunk_size=doc_content_chars_max,
    chunk_overlap=0,
    is_separator_regex=True)

token_splitter = TokenTextSplitter(chunk_size=doc_tokens, chunk_overlap=0)
parent_transformer = DocumentTransformerPipeline(transformers=[wiki_splitter,token_splitter])

As you can see, it looks a lot like Runnable.

Similarly, I propose a DocumentTransformers class to allow transforming a document into multiple versions.

chunk_transformer = DocumentTransformers(
    transformers=[
        GenerateQuestionsTransformer.from_llm(llm),
        SummarizeTransformer.from_llm(llm),
        CopyDocumentTransformer(),
    ]
)

In this case, these transformations can be either synchronous or asynchronous, depending on the context, as they invoke an LLM to generate questions or a summary.

The syntax for invocation is as follows:

ParentDocumentRetriever(
  parent_splitter=DocumentTransformerPipeline(transformers=[wiki_splitter,token_splitter]),
  child_splitter=DocumentTransformers(
    transformers=[
        GenerateQuestionsTransformer.from_llm(llm),
        SummarizeTransformer.from_llm(llm),
        CopyDocumentTransformer(),
    ]
  )
)

It's still quite cumbersome. By declaring all transformations as Runnables, it would be possible to write instead:

ParentDocumentRetriever(
  parent_splitter=wiki_splitter | token_splitter,
  child_splitter=GenerateQuestionsTransformer.from_llm(llm) & 
                SummarizeTransformer.from_llm(llm) &
                CopyDocumentTransformer(),
    ]
  )
)

This is more elegant.

And we return to the initial proposition of making the transformations lazy. Why is this particularly interesting? When invoking index() with RAGVectorstore, index() will be able to identify if a document has not changed, and thus it is not necessary to apply transformations that can be costly in LLM invocations.

By combining these two proposals, it is possible to work with very large lists of documents without saturating memory. Only modified documents will be transformed.

In fact, this allows reversing the classic development model:

• I take the documents.
• I split them.
• I decide if they need to be stored in the vector database.

Instead, I propose:

• I take the loader.
• I store them in the vector database (via index).
• If necessary, and only if necessary, the vectorstore decide to split and transform them.

This approach saves memory, improves performance, reduces LLM invocations, and simplifies the process.

I imagine you're very busy. But, if you have a moment to look at the following notebook, you'll understand the advantages of combining RAGVectorStore and index(). You can imagine how much simpler it would be if you could also use Runnable syntax.

And that's it.

To sum up:

• being lazy during transformations saves memory, cpu, and reduces LLM invocations, while being compatible with the entire Runnable ecosystem.
• Adding RAGVectorStore would make ParentDocumentRetriever compatible with index().
• Combining the two makes it easier to implement advanced RAG projects, while benefiting from all the advanced strategies of Retrievers.

What are your thoughts on this?

PS: Please, can you check this other langchain-ai/langchain-postgres#32 (comment) ? it has been blocked for 3 weeks because "@eyurtsev requested changes", but I can't find the changes I want, so I can't validate them

[pprados]

another syntax that will be possible to have:

result = index(
  docs_source=loader.lazy_load() | wiki_splitter | token_splitter ,
  cleanup="incremental",
  batch_size=100,
  **index_kwargs
)