Duplicate Page content seen with different metadata as a result of space text splitter in linux

[swathithiyan]

Hi

We are using spacy text_splitter to chunk a document and write into chroma DB.
What we see that , post chunking(docs = text_splitter.split_documents(documents)), we are seeing duplicate page contents with different metadata, refer the screenshot, key is the duplicate page content , value is the number of times , its repeated and list of metadata.

what's strange is that , issue is happening in linux OS, in MAC OS it works perfectly.
spacy version used is 3.5.1.
can you pls help here.

[svlandeg]

Hi!

When creating a new issue or discussion topic, it would be beneficial to paste the code you ran (a minimal reproducable example) and to paste examples instead of taking screenshots, which are difficult for us to process.

That said, I assume you're using Langchain's SpacyTextSplitter which has not been created by us.