Does AzureBlobStorageContainerLoader support load markdown file can still reserve markdown format?

[hueiyuan]

Hi, everyone.
I would like to check could we reserve markdown format during using AzureBlobStorageContainerLoader to load markdown file in azure blob storage?
Because we want to use MarkdownHeaderTextSplitter to process so that can ensure paragraph integrity.

If you have any ideas, please share to me.
I will be appreciate it , thank you.

[dosubot[bot]]

🤖

Hi,

Based on the information available in the LangChain repository, the AzureBlobStorageContainerLoader does indeed support loading files from Azure Blob Storage. This is evident from the load method in the AzureBlobStorageContainerLoader class. Here is the relevant code:

def load(self) -> List[Document]:
    """Load documents."""
    try:
        from azure.storage.blob import ContainerClient
    except ImportError as exc:
        raise ImportError(
            "Could not import azure storage blob python package. "
            "Please install it with `pip install azure-storage-blob`."
        ) from exc

    container = ContainerClient.from_connection_string(
        conn_str=self.conn_str, container_name=self.container
    )
    docs = []
    blob_list = container.list_blobs(name_starts_with=self.prefix)
    for blob in blob_list:
        loader = AzureBlobStorageFileLoader(
            self.conn_str, self.container, blob.name  # type: ignore
        )
        docs.extend(loader.load())
    return docs

(Source: libs/langchain/langchain/document_loaders/azure_blob_storage_container.py)

As for preserving the markdown format during the loading process, the UnstructuredMarkdownLoader is used to load markdown files and it does preserve the markdown format. This is done using the partition_md function from the unstructured library. Here is the relevant code:

from unstructured.partition.md import partition_md

return partition_md(filename=self.file_path, **self.unstructured_kwargs)

(Source: libs/langchain/langchain/document_loaders/markdown.py)

However, I wasn't able to find information on whether the AzureBlobStorageContainerLoader utilizes the UnstructuredMarkdownLoader for loading markdown files. It's possible that this information is available elsewhere or I may have missed it.

I hope this information helps! If you have any more questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/document_loaders/markdown.py
• libs/langchain/langchain/document_loaders/azure_blob_storage_container.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.