embedDocuments() batch support and possibly large document splitting

[krodyrobi]

Describe the Problem

OpenAi apis limit the number of elements embedded at a time and the size of each element, to work around this the js sdk and python sdk provide the following:

• batching if number of documents are above 2048 (promise.all on the batch api calls)
• split large (>~8000 tokens) documents in sections and then join the retrieved embeddings in a normalized fashion (here the js sdk does not do it but python one does)

Propose a Solution

Base implementation of sdk chat model on @langchain/oneai models.
For the per document split I believe this needs a custom implementation above the core sdk, didn't see it in js langchain.

Describe Alternatives

Create a wrapper over existing model.

Affected Development Phase

Development

Impact

Inconvenience

Timeline

No response

Additional Context

No response

[ZhongpinWang]

Hi @krodyrobi ,

Thanks for raising this feature request. Have you checked langchain textsplitters, langchain vector store, and sample code? We currently do not handle sending batch requests or join embeddings. Will this satisfy your needs since you are using langchain?

[krodyrobi]

Out of all the feature requests this in my opinion can be treated just as nice to have.

For the split of a single document, realistically no document should be that big to begin with as you've pointed out splitters exist, added here just for feature parity with python gen ai sdk and langchain.

For batching multiple documents however langchain-js/openai does this as a convenience. I realize we should stay clear from using something like this as if during the batch 1 request fails we lose quota for the whole promise.all set. Again this is here to provide parity with existing pre-conditions in js and python sdks.

[jjtang1985]

Hi @krodyrobi ,

Thanks for raising this up.

I just want to point out the grounding module of the orchestration, which supports both embedding and retrieval.
Additionally, the pipeline API allows you to provide e.g., PDF in your Microsoft SharePoint and do the embedding for you.

Best regards,
Junjie

[krodyrobi]

Thank you for the heads up. Unfortunatelly at this point in time the grounding / pipeline api do not provide a clear solution for ingesting documents in a anonymised fashion, for which we have a strict requirement. Both in storing the chunks and for interractions with the embedding 3rd party apis.