Adding a max chunk size with SemanticChunker

[killian-mahe]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Adding a way to limit the size of the chunks created by the SemanticChunker.

Motivation

The models have limits on the number of input tokens and cannot embed chunks that are too large. SemanticChunker has no limit on chunk size, which can lead to errors when chunking or embedding large chunks.

Proposal (If applicable)

Add a max_chunk_size paramter in the SemanticChunker instantiation.

[killian-mahe]

I've created a pull requet for this feature : #18019

[ramkrishna757575]

Please add this feature because otherwise it is very difficult to use SemanticChunker.
Most of the times, it it creating an extremely big document chunk due to which context window is exceeded.

[RafaelXokito]

I'm not sure I understood the suggestion, but I created a PR for it #24342.
Leave your feedback.