How to disable the warnings logs (created a chunk of size... ) that we get during text splitting?

[prakhar7651]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

python 
'''
from langchain.text_splitter import NLTKTextSplitter

splitter = NLTKTextSplitter(chunk_size = 1, chunk_overlap = 0)

text = "This is a test of the langchain module. It is a very simple test, but it should work."

splits = splitter.split_text(text)
'''

Description

If we run the above code, we get some warning logs saying that Created a chunk of size 39, which is longer than the specified 1. Can we disable this logging? We can use the logging module to disable warning logs but should we allow a parameter (like verbose) which can control this behavior?

System Info

langchain==0.0.297
langchain-core==0.1.1

linux

Python 3.10.12

[dosubot[bot]]

Hey there, @prakhar7651! Great to see you diving into the depths of LangChain again. 🚀

Based on the information provided, the NLTKTextSplitter class in the LangChain codebase does not currently support a verbose parameter or similar functionality to control logging behavior. However, the LangChain codebase does provide a global logging configuration that can be adjusted by the user to control the verbosity of logs across all modules.

You can use the set_verbose(value: bool) -> None and set_debug(value: bool) -> None functions to set the verbosity and debug levels globally. Here's an example of how you can use these functions:

from langchain_core.globals import set_verbose, set_debug

# Disable verbose logging
set_verbose(False)

# Disable debug logging
set_debug(False)

This will affect how verbose or detailed the logging output will be across all modules that respect these settings. Please note that this might not suppress the specific warning you mentioned, as it depends on how the NLTKTextSplitter class is implemented.

If you want to suppress all warnings, you can adjust the logging level to ERROR or CRITICAL using the logging module:

import logging

logging.getLogger().setLevel(logging.ERROR)

This will suppress all logs with a level lower than ERROR, including WARNING logs. However, please be aware that this will also suppress other potentially important warnings.

You can find more information about logging in Python in the official Python documentation.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• libs/text-splitters/langchain_text_splitters/nltk.py
• libs/langchain/tests/unit_tests/callbacks/tracers/test_logging.py
• libs/core/langchain_core/globals/init.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[prakhar7651]

@dosu-bot I tried both set_verbose and set_debug, but none of them was able to suppress the warning logs coming from the splitter.

[dosubot[bot]]

Based on the information you've provided, it seems like the warning logs you're seeing are not directly related to the NLTKTextSplitter class or the set_verbose and set_debug functions from the langchain_core.globals module. The warning message Created a chunk of size 39, which is longer than the specified 1 suggests that it might be coming from another part of the LangChain library that handles chunking of text data.

To suppress these specific warning logs, you can use Python's built-in warnings module. Here's how you can modify your code to ignore these warnings:

import warnings
from langchain.text_splitter import NLTKTextSplitter
from langchain_core.globals import set_verbose, set_debug

# Ignore all warnings
warnings.filterwarnings("ignore")

set_verbose(False)
set_debug(False)

splitter = NLTKTextSplitter(chunk_size = 1, chunk_overlap = 0)

text = "This is a test of the langchain module. It is a very simple test, but it should work."

splits = splitter.split_text(text)

This code will suppress all warnings. If you want to suppress a specific category of warnings, you can replace "ignore" with the category of the warning you want to suppress. However, without more information about the specific warning category for the warning log Created a chunk of size 39, which is longer than the specified 1, it's difficult to provide a more targeted solution.

Please note that suppressing warnings can sometimes hide important information about potential issues in your code or the libraries you're using, so use this approach with caution.

If you continue to experience issues or if you have more information about the specific warning category, please provide more details so I can assist you better.

Sources

• libs/core/langchain_core/language_models/base.py
• libs/text-splitters/langchain_text_splitters/nltk.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[prakhar7651]

Disabling logging this way can lead to some other warnings being ignored. Should we rather have a parameter in the split_text method that we can set/unset which controls these warnings logs ?

[yli110-stat697]

for me, even after warnings.filterwarnings("ignore"), I still got such messages

I am using from langchain.text_splitter import CharacterTextSplitter

[prakhar7651]

any updates on this?

[thenicekat]

from langchain.text_splitter import NLTKTextSplitter
import logging

old_log_record_factory = logging.getLogRecordFactory()

def new_log_record_factory(*args, **kwargs):
    record = old_log_record_factory(*args, **kwargs)
    if record.getMessage().startswith("Created a chunk of size"):
        return None
    return record

logging.setLogRecordFactory(new_log_record_factory)


splitter = NLTKTextSplitter(chunk_size = 1, chunk_overlap = 0)

text = "This is a test of the langchain module. It is a very simple test, but it should work."

splits = splitter.split_text(text)
for split in splits:
    print(f"Split: {split} with length {len(split)}")

I think this should work. It ignores LogRecords that pertain to Created a chunk of size
Do let me know if this is not a safe way

[prakhar7651]

Ideally, the splitter module should accept a parameter that would make the module less verbose. Thoughts on this?

[okovalenko88]

I was using CharacterTextSplitter from langchain_text_splitters and the warning messages were in the form "WARNING:langchain_text_splitters.base:Created a chunk of size...". I was able to get rid of them by:

import logging
logging.getLogger("langchain_text_splitters.base").setLevel(logging.ERROR)