Merge branch 'chat-history' of github.com:leila-messallem/neo4j-graphrag-python into chat-history

Author: leila-messallem

CHANGELOG.md

@@ -2,6 +2,15 @@
 
 ## Next
 
+## Added
+- Integrated json-repair package to handle and repair invalid JSON generated by LLMs.
+- Introduced InvalidJSONError exception for handling cases where JSON repair fails.
+- Ability to create a Pipeline or SimpleKGPipeline from a config file. See [the example](examples/build_graph/from_config_files/simple_kg_pipeline_from_config_file.py).
+- Added `OllamaLLM` and `OllamaEmbeddings` classes to make Ollama support more explicit. Implementations using the `OpenAILLM` and `OpenAIEmbeddings` classes will still work.
+
+## Changed
+- Updated LLM prompts to include stricter instructions for generating valid JSON.
+
 ### Fixed
 - Added schema functions to the documentation.
 

README.md

@@ -27,6 +27,33 @@ pip install neo4j-graphrag
 
 ### Optional Dependencies
 
+This package has some optional features that can be enabled using
+the extra dependencies described below:
+
+- LLM providers (at least one is required for RAG and KG Builder Pipeline):
+    - **ollama**: LLMs from Ollama
+    - **openai**: LLMs from OpenAI (including AzureOpenAI)
+    - **google**: LLMs from Vertex AI
+    - **cohere**: LLMs from Cohere
+    - **anthropic**: LLMs from Anthropic
+    - **mistralai**: LLMs from MistralAI
+- **sentence-transformers** : to use embeddings from the `sentence-transformers` Python package
+- Vector database (to use :ref:`External Retrievers`):
+    - **weaviate**: store vectors in Weaviate
+    - **pinecone**: store vectors in Pinecone
+    - **qdrant**: store vectors in Qdrant
+- **experimental**: experimental features such as the Knowledge Graph creation pipelines.
+    - Warning: this dependency group requires `pygraphviz`. See below for installation instructions.
+
+
+Install package with optional dependencies with (for instance):
+
+```shell
+pip install "neo4j-graphrag[openai]"
+# or
+pip install "neo4j-graphrag[openai, experimental]"
+```
+
 #### pygraphviz
 
 `pygraphviz` is used for visualizing pipelines.

docs/source/api.rst

@@ -9,17 +9,18 @@ API Documentation
 Components
 **********
 
-KGWriter
-========
+DataLoader
+==========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
-    :members: run
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.DataLoader
+    :members: run, get_document_metadata
 
-Neo4jWriter
-===========
 
-.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
-    :members: run
+PdfLoader
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.components.pdf_loader.PdfLoader
+    :members: run, load_file
 
 TextSplitter
 ============
@@ -85,6 +86,17 @@ LLMEntityRelationExtractor
 .. autoclass:: neo4j_graphrag.experimental.components.entity_relation_extractor.LLMEntityRelationExtractor
     :members: run
 
+KGWriter
+========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.KGWriter
+    :members: run
+
+Neo4jWriter
+===========
+
+.. autoclass:: neo4j_graphrag.experimental.components.kg_writer.Neo4jWriter
+    :members: run
 
 SinglePropertyExactMatchResolver
 ================================
@@ -112,6 +124,23 @@ SimpleKGPipeline
     :members: run_async
 
 
+************
+Config files
+************
+
+
+SimpleKGPipelineConfig
+======================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.template_pipeline.simple_kg_builder.SimpleKGPipelineConfig
+
+
+PipelineRunner
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.runner.PipelineRunner
+
+
 .. _retrievers-section:
 
 **********
@@ -210,6 +239,12 @@ AzureOpenAIEmbeddings
 .. autoclass:: neo4j_graphrag.embeddings.openai.AzureOpenAIEmbeddings
     :members:
 
+OllamaEmbeddings
+================
+
+.. autoclass:: neo4j_graphrag.embeddings.ollama.OllamaEmbeddings
+    :members:
+
 VertexAIEmbeddings
 ==================
 
@@ -257,6 +292,12 @@ AzureOpenAILLM
     :members:
     :undoc-members: get_messages, client_class, async_client_class
 
+OllamaLLM
+---------
+
+.. autoclass:: neo4j_graphrag.llm.ollama_llm.OllamaLLM
+    :members:
+
 
 VertexAILLM
 -----------
@@ -297,12 +338,21 @@ RagTemplate
 
 .. autoclass:: neo4j_graphrag.generation.prompts.RagTemplate
     :members:
+    :exclude-members: format
 
 ERExtractionTemplate
 --------------------
 
 .. autoclass:: neo4j_graphrag.generation.prompts.ERExtractionTemplate
     :members:
+    :exclude-members: format
+
+Text2CypherTemplate
+--------------------
+
+.. autoclass:: neo4j_graphrag.generation.prompts.Text2CypherTemplate
+    :members:
+    :exclude-members: format
 
 
 ****
@@ -384,6 +434,8 @@ Errors
 
   * :class:`neo4j_graphrag.experimental.pipeline.exceptions.PipelineStatusUpdateError`
 
+  * :class:`neo4j_graphrag.experimental.pipeline.exceptions.InvalidJSONError`
+
 
 Neo4jGraphRagError
 ==================
@@ -414,7 +466,7 @@ FilterValidationError
 
 
 EmbeddingsGenerationError
-========================
+=========================
 
 .. autoclass:: neo4j_graphrag.exceptions.EmbeddingsGenerationError
    :show-inheritance:
@@ -509,3 +561,10 @@ PipelineStatusUpdateError
 
 .. autoclass:: neo4j_graphrag.experimental.pipeline.exceptions.PipelineStatusUpdateError
    :show-inheritance:
+
+
+InvalidJSONError
+================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.exceptions.InvalidJSONError
+   :show-inheritance:

docs/source/images/kg_builder_pipeline.png

@@ -72,6 +72,37 @@ To install the latest stable version, use:
 
    It is always recommended to install python packages for user space in a virtual environment.
 
+*********************
+Optional Dependencies
+*********************
+
+Extra dependencies can be installed with:
+
+.. code:: bash
+
+    pip install "neo4j-graphrag[openai]"
+    # or
+    pip install "neo4j-graphrag[openai, experimental]"
+
+
+List of extra dependencies:
+
+- LLM providers (at least one is required for RAG and KG Builder Pipeline):
+    - **ollama**: LLMs from Ollama
+    - **openai**: LLMs from OpenAI (including AzureOpenAI)
+    - **google**: LLMs from Vertex AI
+    - **cohere**: LLMs from Cohere
+    - **anthropic**: LLMs from Anthropic
+    - **mistralai**: LLMs from MistralAI
+- **sentence-transformers** : to use embeddings from the `sentence-transformers` Python package
+- Vector database (to use :ref:`External Retrievers`):
+    - **weaviate**: store vectors in Weaviate
+    - **pinecone**: store vectors in Pinecone
+    - **qdrant**: store vectors in Qdrant
+- **experimental**: experimental features such as the Knowledge Graph creation pipelines.
+    - Warning: this requires `pygraphviz`. Installation instructions can be found `here <https://pygraphviz.github.io/documentation/stable/install.html>`_.
+
+
 ********
 Examples
 ********

docs/source/index.rst

@@ -82,3 +82,62 @@ SchemaConfig
 ============
 
 .. autoclass:: neo4j_graphrag.experimental.components.schema.SchemaConfig
+
+LexicalGraphConfig
+===================
+
+.. autoclass:: neo4j_graphrag.experimental.components.types.LexicalGraphConfig
+
+
+Neo4jDriverType
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverType
+
+
+Neo4jDriverConfig
+=================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.Neo4jDriverConfig
+
+
+LLMType
+=======
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMType
+
+
+LLMConfig
+=========
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.LLMConfig
+
+
+EmbedderType
+============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderType
+
+
+EmbedderConfig
+==============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.EmbedderConfig
+
+
+ComponentType
+=============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentType
+
+
+ComponentConfig
+===============
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.object_config.ComponentConfig
+
+
+ParamFromEnvConfig
+==================
+
+.. autoclass:: neo4j_graphrag.experimental.pipeline.config.param_resolver.ParamFromEnvConfig