Text truncation in EmbeddingPipeline due to oversized paragraphs

[Silvio-Re]

I observed the following log messages repeatedly :

(3)(+0009163): [EmbeddingPipeline] [INFO] Text truncated for embedding - {"originalLength":32929,"truncatedLength":8000}

My hypothesis, seeing the "Never splits a paragraph across chunks" chunking heuristic, is that some of my documents have very large text part without line breaks and the text retrieved looks like an unique very long paragraph.
Maybe a fallback in this case could be to split paragraphs longer than the maximum context length of the embedder (8000) at a phrase boundary (Strategy 3: Fixed-size sliding window chunking in https://github.com/introfini/ZotSeek/blob/main/src/utils/chunker.ts ?)