Expand the functionality of the DocumentCleaner

[bilgeyucel]

Is your feature request related to a problem? Please describe.
We've found in practice that cleaning up files before being used in RAG pipelines does increase overall performance. For example, this Haystack user found the same.

We do have a DocumentCleaner to help with this process, but we found there are some options missing for the type of cleaning we would like to accomplish.

Describe the solution you'd like
The options I'd like to add to the DocumentCleaner are:

• an option that just runs .strip() on the content of every document. Often times we just want to remove the extra leading and trailing white space, but leave the white space within a chunk alone. For example, in mark down files the extra newlines can matter for formatting.
• also an option to provide a regex pattern to remove and a string to replace that regex match with. We currently have a few regex replaces in the DocumentCleaner and have the remove_regex parameter, but we don't have a way to customize what string should be used to replace the regex match. For example, one scenario that I'd like to do is replace all double newline characters \n\n with a single newline character \n.

Describe alternatives you've considered
We can create a custom component do perform these operations instead.

[bilgeyucel]

There was a slightly related question on GitHub about passing multiple regular expressions to remove_regex: deepset-ai#8899

[bilgeyucel]

I've looked at the current implementation of the DocumentCleaner component, and I agree that both of your suggested features would be valuable additions.

Here's how I would implement these changes:

1. For the .strip() functionality:

def __init__(
    self,
    remove_empty_lines: bool = True,
    remove_extra_whitespaces: bool = True,
    remove_repeated_substrings: bool = False,
    keep_id: bool = False,
    remove_substrings: Optional[List[str]] = None,
    remove_regex: Optional[str] = None,
    unicode_normalization: Optional[Literal["NFC", "NFKC", "NFD", "NFKD"]] = None,
    ascii_only: bool = False,
    strip_only: bool = False,  # New parameter
):
    # ... existing code ...
    self.strip_only = strip_only

Then in the run method:

def run(self, documents: List[Document]):
    # ... existing validation code ...

    cleaned_docs = []
    for doc in documents:
        if doc.content is None:
            # ... existing code ...
            continue
        text = doc.content

        if self.strip_only:
            # Only strip leading and trailing whitespace
            text = text.strip()
        else:
            # Apply all other cleaning operations
            if self.unicode_normalization:
                text = self._normalize_unicode(text, self.unicode_normalization)
            # ... rest of existing cleaning operations ...
        
        # ... rest of the method ...

2. For the regex replacement functionality, I would modify the __init__ and _remove_regex methods:

def __init__(
    self,
    # ... existing parameters ...
    remove_regex: Optional[str] = None,
    regex_replacement: str = "",  # New parameter with empty string as default
    # ... rest of parameters ...
):
    # ... existing code ...
    self.remove_regex = remove_regex
    self.regex_replacement = regex_replacement
    # ... rest of the method ...

def _remove_regex(self, text: str, regex: str, replacement: str = "") -> str:
    """
    Replace substrings that match the specified regex with the replacement string.

    :param text: Text to clean.
    :param regex: Regex to match substrings.
    :param replacement: String to replace the matched substrings with (default: "").
    :returns: The text with regex matches replaced.
    """
    texts = text.split("\f")
    cleaned_text = [re.sub(regex, replacement, text).strip() for text in texts]
    return "\f".join(cleaned_text)

And then update the call in the run method:

if self.remove_regex:
    text = self._remove_regex(text, self.remove_regex, self.regex_replacement)

These changes would allow users to:

1. Use the strip_only=True parameter to only strip leading and trailing whitespace
2. Use remove_regex="pattern" with regex_replacement="replacement" to replace regex matches with a custom string instead of just removing them

This implementation maintains backward compatibility since both new parameters have default values that match the current behavior.

[haystack-demo-user]

Thanks for reporting this issue! We'll look into it.

[haystack-demo-user]

Thank you for your detailed feature request!

We appreciate your insights on the importance of file cleaning for RAG pipelines and the current limitations you've encountered with the DocumentCleaner. Your suggestions for adding a .strip() option and a customizable regex replacement feature are valuable.

To proceed, we will review the feasibility of implementing these features and consider them for upcoming releases. In the meantime, if you have any specific use cases or examples that could help us understand your needs better, please feel free to share them.

Thank you for your contribution to improving our project!