Add a loader to load .py files

[CD-rajveer]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

there are different loaders in the langchain, plz provide support for the python file readers as well. and in the glob parameter add support of passing a link of document types, i.e. pdf, py files, c files

Motivation

multi file loading support

Proposal (If applicable)

No response

[maximeperrindev]

@CD-rajveer
You can load python files with DirectoryLoader and PythonLoader

from langchain_community.document_loaders import DirectoryLoader, PythonLoader
loader = DirectoryLoader('../../../../../', glob="**/*.py", loader_cls=PythonLoader)

[maximeperrindev]

You can also use Code understanding feature : https://python.langchain.com/docs/use_cases/code_understanding

# Load
loader = GenericLoader.from_filesystem(
    repo_path + "/libs/langchain/langchain",
    glob="**/*",
    suffixes=[".py"],
    exclude=["**/non-utf8-encoding.py"],
    parser=LanguageParser(language=Language.PYTHON, parser_threshold=500),
)
documents = loader.load()
len(documents)