Efficiently rename header handling in ExperimentalMarkdownSyntaxTextSplitter for improved LLM understanding

[david101-hunter]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

DEFAULT_HEADER_KEYS = {
    "#": "Header 1",
    "##": "Header 2",
    "###": "Header 3",
    "####": "Header 4",
    "#####": "Header 5",
    "######": "Header 6",
}

Description

Background

I'm working with the ExperimentalMarkdownSyntaxTextSplitter from the LangChain library to prepare text for processing by an LLM (Language Model). I want to improve how the splitter handles Markdown headers (h1 to h6) to enhance the LLM's understanding of document structure.

In practice, I've found through experimentation that it's not very good yet, as it often provides incomplete answers when the response requires multiple steps,...

Current Approach

Currently, I'm using the default ExperimentalMarkdownSyntaxTextSplitter without any customizatio. In markdown.py of langchain_text_splliters (of langchain library).

DEFAULT_HEADER_KEYS = {
    "#": "Header 1",
    "##": "Header 2",
    "###": "Header 3",
    "####": "Header 4",
    "#####": "Header 5",
    "######": "Header 6",
}

Desired Outcome

I want to refine the splitter to better recognize and handle headers from h1 to h6, potentially by customizing the separator patterns and I also want to know any ways to achieve improving llm answers.

Question

How can I modify the ExperimentalMarkdownSyntaxTextSplitter to improve its handling of Markdown headers (h1 to h6)? Specifically:

1. What custom separator patterns should I use to effectively capture all header levels?
2. Are there any other configuration options I should consider to enhance the splitting process for better LLM understanding?
3. How can I verify that the new splitting method is actually improving the document structure for the LLM?
4. How should we prompt the LLM to focus on these changes?

System Info

Additional Information

• LangChain version: 0.3.1
• Python version: 3.11.9
• Operating System: Ubuntu 20.04