langchain RecursiveCharacterTextSplitter exceeded the specified chunk size

[bzr1]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

RecursiveCharacterTextSplitter(chunk_size=2300, chunk_overlap=50,separators="===========")

Description

some how there is a chunk sized 70000, which is far off. how do I make it: first to respect the separators, then if the chunk is too big, divide based on chunk_size then.

System Info

python version 10

[dosubot[bot]]

Hey @bzr1! I'm Dosu, a bot here to help you with bugs, questions, and becoming a contributor. I'm here to assist you while you wait for a human maintainer.

To ensure that the RecursiveCharacterTextSplitter respects the separators and then divides the text based on the specified chunk_size, you can follow the implementation provided in the RecursiveCharacterTextSplitter class. The class recursively tries to split the text by different characters (separators) and then merges the splits if they are smaller than the specified chunk_size.

Here is the relevant code snippet from the RecursiveCharacterTextSplitter class that demonstrates this behavior:

class RecursiveCharacterTextSplitter(TextSplitter):
    """Splitting text by recursively look at characters.

    Recursively tries to split by different characters to find one
    that works.
    """

    def __init__(
        self,
        separators: Optional[List[str]] = None,
        keep_separator: bool = True,
        is_separator_regex: bool = False,
        **kwargs: Any,
    ) -> None:
        """Create a new TextSplitter."""
        super().__init__(keep_separator=keep_separator, **kwargs)
        self._separators = separators or ["\n\n", "\n", " ", ""]
        self._is_separator_regex = is_separator_regex

    def _split_text(self, text: str, separators: List[str]) -> List[str]:
        """Split incoming text and return chunks."""
        final_chunks = []
        # Get appropriate separator to use
        separator = separators[-1]
        new_separators = []
        for i, _s in enumerate(separators):
            _separator = _s if self._is_separator_regex else re.escape(_s)
            if _s == "":
                separator = _s
                break
            if re.search(_separator, text):
                separator = _s
                new_separators = separators[i + 1 :]
                break

        _separator = separator if self._is_separator_regex else re.escape(separator)
        splits = _split_text_with_regex(text, _separator, self._keep_separator)

        # Now go merging things, recursively splitting longer texts.
        _good_splits = []
        _separator = "" if self._keep_separator else separator
        for s in splits:
            if self._length_function(s) < self._chunk_size:
                _good_splits.append(s)
            else:
                if _good_splits:
                    merged_text = self._merge_splits(_good_splits, _separator)
                    final_chunks.extend(merged_text)
                    _good_splits = []
                if not new_separators:
                    final_chunks.append(s)
                else:
                    other_info = self._split_text(s, new_separators)
                    final_chunks.extend(other_info)
        if _good_splits:
            merged_text = self._merge_splits(_good_splits, _separator)
            final_chunks.extend(merged_text)
        return final_chunks

    def split_text(self, text: str) -> List[str]:
        return self._split_text(text, self._separators)

This code ensures that the text is split using the specified separators and then further divided into chunks based on the chunk_size if necessary. The _split_text method handles the recursive splitting and merging of text chunks.

Additionally, the RecursiveCharacterTextSplitter is parameterized by a list of characters and tries to split on them in order until the chunks are small enough. The default list is ["\n\n", "\n", " ", ""], which helps to keep paragraphs, sentences, and words together as long as possible. You can also customize the list of separators to include additional punctuation or special characters, especially for languages without word boundaries.

Here is an example of how to use the RecursiveCharacterTextSplitter:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load example document
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=20,
    length_function=len,
    is_separator_regex=False,
)
texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
print(texts[1])

# To split text directly
split_texts = text_splitter.split_text(state_of_the_union)
print(split_texts[:2])

This example demonstrates setting a small chunk_size to show how the text is split and how to use the split_text method to obtain the string content directly ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

Your problem can occur if the separators are not present in the text or if the text is not being split correctly by the specified separators. The RecursiveCharacterTextSplitter should respect the separators and then divide the text based on the specified chunk_size.

To ensure this behavior, you can verify the following:

1. Check Separators: Ensure that the separators you are using are present in the text. If the separator is not found, the text won't be split as expected.

2. Custom Separators: If you are using custom separators, make sure they are correctly defined and present in the text. For example, if you are using "===========" as a separator, ensure that this exact string is in the text.

3. Implementation: Use the provided implementation to handle the splitting and merging of text chunks. Here is a simplified example:

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2300,
    chunk_overlap=50,
    separators=["==========="],  # Ensure this separator is present in your text
    keep_separator=True,
    is_separator_regex=False
)

text = "Your text here with =========== as separators"
split_texts = text_splitter.split_text(text)
for chunk in split_texts:
    print(f"Chunk size: {len(chunk)}")
    print(chunk)

This should help ensure that the text is split correctly based on the specified separators and chunk size. If the issue persists, double-check the text and separators to ensure they match your expectations.

To continue talking to Dosu, mention @dosu.

[bzr1]

@dosu im splitting documents tough, with unstructured docx loader

[dosubot[bot]]

To ensure that the RecursiveCharacterTextSplitter respects the specified separators and chunk size when splitting text loaded with the unstructured docx loader, you need to initialize the RecursiveCharacterTextSplitter with the desired settings and then use it to split the loaded text. Here is an example:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Define your custom separators and chunk size
separators = ["==========="]  # Ensure this separator is present in your text
chunk_size = 2300  # Example chunk size

# Initialize the RecursiveCharacterTextSplitter with the custom settings
text_splitter = RecursiveCharacterTextSplitter(
    separators=separators,
    chunk_size=chunk_size,
    chunk_overlap=50,  # Optional: overlap between chunks
    length_function=len,  # Optional: function to determine chunk size
    is_separator_regex=False  # Optional: whether separators are regex
)

# Load your text using the unstructured docx loader
# Assuming `load_docx` is a function that loads the text from a docx file
text = load_docx("path_to_your_docx_file.docx")

# Split the text using the text splitter
chunks = text_splitter.split_text(text)

# Print the chunk sizes to verify
for chunk in chunks:
    print(f"Chunk size: {len(chunk)}")
    print(chunk)

This example demonstrates how to set up the RecursiveCharacterTextSplitter with custom separators and chunk size, load the text from a DOCX file, and then split the text into chunks that respect these settings ^[1][2].

To continue talking to Dosu, mention @dosu.

[bzr1]

@dosu does RecursiveCharacterTextSplitter have a function called split_documents ? I didn't find it in source code, but I can invoke it

[dosubot[bot]]

No, the RecursiveCharacterTextSplitter does not have a function called split_documents. It has a method called split_text which you can use to split the text into chunks. If you need to split documents, you should first load the document's content into a string and then use the split_text method to split that string into chunks.

To continue talking to Dosu, mention @dosu.

[wy371900521]

text_spliter = CharacterTextSplitter(
separator=',',
chunk_size=10,
chunk_overlap=2
)
result = text_spliter.split_text("今天天气好晴朗,处处好风光啊好风光蝴蝶儿忙啊,蜜蜂也忙,小鸟儿忙着,白云也忙")
print(result) # => ['今天天气好晴朗', ',处处好风光啊好风光蝴蝶儿忙啊', ',蜜蜂也忙', ',小鸟儿忙着', ',白云也忙']
chunk_overlap 没有起到作用为啥?