Why UnstructuredMarkdownLoader strips out formatting ?

[paolomainardi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.document_loaders import UnstructuredMarkdownLoader

loader = UnstructuredMarkdownLoader(
    "./example_data/example.md",
    mode="single",
    strategy="fast",
)

Description

The issue is that I am using DirectoryLoader with or without loader_cls on a directory containing markdown files and the result is that the parsed content is basically just raw text, all formatting gets deleted.

Is this the right behaviour ? If yes, if i want to keep markdown formatting to be used in a RAG applications, what should i use instead ?

System Info

(.venv) vscode ➜ $ pip freeze | grep langchain
langchain==0.3.0
langchain-anthropic==0.2.1
langchain-chroma==0.1.4
langchain-cli==0.0.31
langchain-community==0.3.0
langchain-core==0.3.5
langchain-openai==0.2.0
langchain-text-splitters==0.3.0
langchain-unstructured==0.1.4

(.venv) vscode ➜ $ pip freeze | grep langchain
langchain-unstructured==0.1.4
unstructured==0.15.13
unstructured-client==0.25.9
unstructured-inference==0.7.36
unstructured.pytesseract==0.3.13

[dosubot[bot]]

UnstructuredMarkdownLoader strips out formatting by default because it combines different "elements" of the markdown into a single document. This is the intended behavior for basic usage.

To keep markdown formatting for RAG (Retrieval-Augmented Generation) applications, you should use the mode="elements" option when initializing the UnstructuredMarkdownLoader. This will retain the separation of different elements in the markdown.

Here is how you can do it:

from langchain_community.document_loaders import UnstructuredMarkdownLoader

markdown_path = "./example_data/example.md"
loader = UnstructuredMarkdownLoader(markdown_path, mode="elements")

data = loader.load()
print(f"Number of documents: {len(data)}\n")

for document in data[:2]:
    print(f"{document}\n")

Additionally, if you are using the DirectoryLoader, you can configure it to use UnstructuredMarkdownLoader:

from langchain_community.document_loaders import DirectoryLoader
from langchain_community.document_loaders.unstructured_file import UnstructuredMarkdownLoader

loader = DirectoryLoader("./example_data", glob="**/*.md", loader_cls=UnstructuredMarkdownLoader)
docs = loader.load()

This configuration ensures that DirectoryLoader uses UnstructuredMarkdownLoader to parse Markdown files while preserving the formatting ^[1][2].

To continue talking to Dosu, mention @dosu.

[paolomainardi]

And how can I pass this argument when using DirectoryLoader ? It seems that i just pass the loader class.

[dosubot[bot]]

To pass the mode="elements" argument when using DirectoryLoader with UnstructuredMarkdownLoader, you can initialize the UnstructuredMarkdownLoader with the mode="elements" argument as shown in the example below:

from langchain_community.document_loaders import DirectoryLoader
from langchain_community.document_loaders.unstructured_file import UnstructuredMarkdownLoader

loader = DirectoryLoader(
    "./example_data",
    glob="**/*.md",
    loader_cls=UnstructuredMarkdownLoader,
    loader_kwargs={"mode": "elements"}
)

docs = loader.load()

This configuration ensures that DirectoryLoader uses UnstructuredMarkdownLoader to parse Markdown files while preserving the formatting by retaining the elements ^[1].

To continue talking to Dosu, mention @dosu.

[kosson]

Thanks @paolomainardi for digging this up. It really helped. My updated code:

class DocumentManager:
    def __init__(self, directory_path, glob_pattern="./**/*.md"):
        self.directory_path = directory_path
        self.glob_pattern = glob_pattern
        self.documents = []
        self.all_sections = []
    
    def load_documents(self):
        loader = DirectoryLoader(self.directory_path, glob=self.glob_pattern, show_progress=True, loader_cls=UnstructuredMarkdownLoader, loader_kwargs={"mode": "elements"})
        self.documents = loader.load()