Pipeline from config file (#220)

* SimpleKGPipeline config parser

* Factorize common parmeters

* Fix

* Fix again (ruff)

* Add builder, docstrings

* Adds example

* Ruff

* Add headers

* Another header

* Remove old file - more detailed example

* Fix JSON

* WIP

* Adds more param resolvers

* A bit of mypy

* Add root types to allow instantiation from python object directly

* Document + mypy

* Add embedder config

* Implement SimpleKGBuilder with this setup

* Add YAML config example

* Example config files for custom pipeline

* Update SimpleKGPipeline

* Fix UT

* Fix tests

* Simplify and mypy

* ruff

* Add missing dep

* Missing import for '|' annotations

* Restructure files + increase tests coverage

* mypy

* Refactor examples

* Test runner, clean simple kg builder test (remove duplicates)

* E2E tests

* Use fsspec in config reader

* Changelog

* Fix test

* Use cast to remove a type ignore comment

* Also use cast here to remove type ignore

* Close instantiated drivers

* ruff

* Adding loggers

* Make close function async

* Fix tests

* fix UT

* Write doc about SimpleKGPipeline and config files

* Update api.rst

* Recreate lock file after merge

* Add more comments to explain Config/Type models

Author: stellasia

CHANGELOG.md

@@ -5,6 +5,7 @@
 ## Added
 - Integrated json-repair package to handle and repair invalid JSON generated by LLMs.
 - Introduced InvalidJSONError exception for handling cases where JSON repair fails.
+- Ability to create a Pipeline or SimpleKGPipeline from a config file. See [the example](examples/build_graph/from_config_files/simple_kg_pipeline_from_config_file.py).
 
 ## Changed
 - Updated LLM prompts to include stricter instructions for generating valid JSON.

docs/source/api.rst

@@ -9,17 +9,18 @@ API Documentation
 Components
 **********
 
-KGWriter
-========
+DataLoader
+==========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
-    :members: run
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.DataLoader
+    :members: run, get_document_metadata
 
-Neo4jWriter
-===========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
-    :members: run
+PdfLoader
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.PdfLoader
+    :members: run, load_file
 
 TextSplitter
 ============
@@ -85,6 +86,17 @@ LLMEntityRelationExtractor
 .. autoclass:: neo4j_graphrag.experimental.components.entity_relation_extractor.LLMEntityRelationExtractor
     :members: run
 
+KGWriter
+========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
+    :members: run
+
+Neo4jWriter
+===========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
+    :members: run
 
 SinglePropertyExactMatchResolver
 ================================
@@ -112,6 +124,23 @@ SimpleKGPipeline
     :members: run_async
 
 
+************
+Config files
+************
+
+
+SimpleKGPipelineConfig
+======================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.template_pipeline.simple_kg_builder.SimpleKGPipelineConfig
+
+
+PipelineRunner
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.runner.PipelineRunner
+
+
 .. _retrievers-section:
 
 **********

docs/source/images/kg_builder_pipeline.png

@@ -82,3 +82,62 @@ SchemaConfig
 ============
 
 .. autoclass:: neo4j_graphrag.experimental.components.schema.SchemaConfig
+
+LexicalGraphConfig
+===================
+
+.. autoclass:: neo4j_graphrag.experimental.components.types.LexicalGraphConfig
+
+
+Neo4jDriverType
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverType
+
+
+Neo4jDriverConfig
+=================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverConfig
+
+
+LLMType
+=======
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMType
+
+
+LLMConfig
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMConfig
+
+
+EmbedderType
+============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderType
+
+
+EmbedderConfig
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderConfig
+
+
+ComponentType
+=============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentType
+
+
+ComponentConfig
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentConfig
+
+
+ParamFromEnvConfig
+==================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.param_resolver.ParamFromEnvConfig