feat: Add support for Markdown files (#496)

Author: ali-sedaghatbaf

CHANGELOG.md

@@ -2,11 +2,21 @@
 
 ## Next
 
+### Added
+
+- MarkdownLoader (experimental): added a Markdown loader to support `.md` and `.markdown` files.
+
+### Changed
+
+- SimpleKG pipeline (experimental): the `from_pdf` parameter is deprecated in favor of `from_file` (PDF and Markdown inputs). `from_pdf` still works but emits a deprecation warning and will be removed in a future version.
+- Data loaders (experimental): the `PdfDocument` type name is deprecated in favor of `LoadedDocument`; `PdfDocument` remains available as a backward-compatible alias with a deprecation warning.
+
 ## 1.14.1
 
 ### Added
 
 - `NodeType` and `RelationshipType` now reject labels and types that start or end with double underscores (`__`), e.g. `__Person__`. This convention is reserved for internal Neo4j GraphRAG labels. A `ValidationError` is raised on construction.
+- SimpleKG pipeline (experimental): Markdown inputs (`.md` / `.markdown`) are supported alongside PDF via the default extension-based file loader when building from a file path.
 
 ### Changed
 

README.md

@@ -138,7 +138,7 @@ kg_builder = SimpleKGPipeline(
         "patterns": patterns,
     },
     on_error="IGNORE",
-    from_pdf=False,
+    from_file=False,
 )
 
 # Run the pipeline on a piece of text

docs/source/api.rst

@@ -18,13 +18,19 @@ Component
 DataLoader
 ==========
 
-.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.DataLoader
+.. autoclass:: neo4j_graphrag.experimental.components.data_loader.DataLoader
     :members: run, get_document_metadata
 
 PdfLoader
 =========
 
-.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.PdfLoader
+.. autoclass:: neo4j_graphrag.experimental.components.data_loader.PdfLoader
+    :members: run, load_file
+
+MarkdownLoader
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.components.data_loader.MarkdownLoader
     :members: run, load_file
 
 TextSplitter

docs/source/types.rst

@@ -44,6 +44,11 @@ DocumentInfo
 
 .. autoclass:: neo4j_graphrag.experimental.components.types.DocumentInfo
 
+LoadedDocument
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.components.types.LoadedDocument
+
 
 TextChunk
 =========

docs/source/user_guide_kg_builder.rst

@@ -54,10 +54,10 @@ is utilizing the `SimpleKGPipeline` interface:
         llm=llm, # an LLMInterface for Entity and Relation extraction
         driver=neo4j_driver,  # a neo4j driver to write results to graph
         embedder=embedder,  # an Embedder for chunks
-        from_pdf=True,   # set to False if parsing an already extracted text
+        from_file=True,   # set to False if parsing an already extracted text
     )
     await kg_builder.run_async(file_path=str(file_path))
-    # await kg_builder.run_async(text="my text")  # if using from_pdf=False
+    # await kg_builder.run_async(text="my text")  # if using from_file=False
 
 
 See:
@@ -216,9 +216,12 @@ instances of specific components to the `SimpleKGPipeline`. The components that
 customized at the moment are:
 
 - `text_splitter`: must be an instance of :ref:`TextSplitter`
-- `pdf_loader`: must be an instance of :ref:`PdfLoader`
+- `file_loader`: must be an instance of :ref:`PdfLoader` or :ref:`MarkdownLoader`
 - `kg_writer`: must be an instance of :ref:`KGWriter`
 
+The legacy names ``from_pdf`` and ``pdf_loader`` (in Python, YAML, or JSON) are still accepted
+with a deprecation warning; use ``from_file`` and ``file_loader`` instead.
+
 For instance, the following code can be used to customize the chunk size and
 chunk overlap in the text splitter component:
 
@@ -450,7 +453,7 @@ within the configuration file.
 .. code:: json
 
     {
-        "from_pdf": false,
+        "from_file": false,
         "perform_entity_resolution": true,
         "neo4j_database": "myDb",
         "on_error": "IGNORE",
@@ -502,7 +505,7 @@ or in YAML:
 
 .. code:: yaml
 
-    from_pdf: false
+    from_file: false
     perform_entity_resolution: true
     neo4j_database: myDb
     on_error: IGNORE
@@ -578,7 +581,7 @@ Each of these components can be run individually:
 .. code:: python
 
     import asyncio
-    from neo4j_graphrag.experimental.components.pdf_loader import PdfLoader
+    from neo4j_graphrag.experimental.components.data_loader import PdfLoader
     my_component = PdfLoader()
     asyncio.run(my_component.run("my_file.pdf"))
 
@@ -588,7 +591,7 @@ They can also be used within a pipeline:
 .. code:: python
 
     from neo4j_graphrag.experimental.pipeline import Pipeline
-    from neo4j_graphrag.experimental.components.pdf_loader import PdfLoader
+    from neo4j_graphrag.experimental.components.data_loader import PdfLoader
     pipeline = Pipeline()
     my_component = PdfLoader()
     pipeline.add_component(my_component, "component_name")
@@ -604,7 +607,7 @@ This package currently supports text extraction from PDFs:
 .. code:: python
 
     from pathlib import Path
-    from neo4j_graphrag.experimental.components.pdf_loader import PdfLoader
+    from neo4j_graphrag.experimental.components.data_loader import PdfLoader
 
     loader = PdfLoader()
     await loader.run(filepath=Path("my_file.pdf"))
@@ -614,12 +617,13 @@ To implement your own loader, use the `DataLoader` interface:
 .. code:: python
 
     from pathlib import Path
-    from neo4j_graphrag.experimental.components.pdf_loader import DataLoader, PdfDocument
+    from neo4j_graphrag.experimental.components.data_loader import DataLoader
+    from neo4j_graphrag.experimental.components.types import LoadedDocument
 
     class MyDataLoader(DataLoader):
-        async def run(self, filepath: Path, metadata: Optional[Dict[str, str]] = None) -> PdfDocument:
+        async def run(self, filepath: Path, metadata: Optional[Dict[str, str]] = None) -> LoadedDocument:
             # process file in `filepath`
-            return PdfDocument(
+            return LoadedDocument(
                 text="text",
                 document_info=DocumentInfo(
                     path=str(filepath),

examples/build_graph/automatic_schema_extraction/simple_kg_builder_schema_from_pdf.py

@@ -54,7 +54,7 @@ async def run_kg_pipeline_with_auto_schema() -> None:
             llm=llm,
             driver=driver,
             embedder=embedder,
-            from_pdf=True,
+            from_file=True,
         )
 
         print(f"Processing PDF file: {PDF_FILE}")

examples/build_graph/automatic_schema_extraction/simple_kg_builder_schema_from_text.py

@@ -73,7 +73,7 @@ async def run_kg_pipeline_with_auto_schema() -> None:
             llm=llm,
             driver=driver,
             embedder=embedder,
-            from_pdf=False,  # Using raw text input, not PDF
+            from_file=False,  # Using raw text input, not PDF
         )
 
         # Run the pipeline on the text

examples/build_graph/from_config_files/simple_kg_pipeline_config.json

@@ -36,7 +36,7 @@
             }
         }
     },
-    "from_pdf": false,
+    "from_file": false,
     "schema": {
         "node_types": [
             "Person",

examples/build_graph/from_config_files/simple_kg_pipeline_config.yaml

@@ -24,7 +24,7 @@ embedder_config:
     api_key:
       resolver_: ENV
       var_: OPENAI_API_KEY
-from_pdf: false
+from_file: false
 schema:
   node_types:
     - label: Person

examples/build_graph/from_config_files/simple_kg_pipeline_config_url.json

@@ -36,7 +36,7 @@
             }
         }
     },
-    "from_pdf": true,
+    "from_file": true,
     "schema": {
         "node_types": [
             "Person",
@@ -105,8 +105,8 @@
             "chunk_overlap": 10
         }
     },
-    "pdf_loader": {
-        "class_": "pdf_loader.PdfLoader",
+    "file_loader": {
+        "class_": "data_loader.PdfLoader",
         "run_params_": {
             "fs": "http"
         }