Merge branch 'main' into improve-llm-generated-json-fixing

Author: willtai

CHANGELOG.md

@@ -2,17 +2,37 @@
 
 ## Next
 
+## 1.2.1
+
+### Added
+- Introduced optional lexical graph configuration for `SimpleKGPipeline`, enhancing flexibility in customizing node labels and relationship types in the lexical graph.
+- Introduced optional `neo4j_database` parameter for `SimpleKGPipeline`, `Neo4jChunkReader`and `Text2CypherRetriever`.
+- Ability to provide description and list of properties for entities and relations in the `SimpleKGPipeline` constructor.
+
+### Fixed
+- `neo4j_database` parameter is now used for all queries in the `Neo4jWriter`.
+
+### Changed
+- Updated all examples to use `neo4j_database` parameter instead of an undocumented neo4j driver constructor.
+- All `READ` queries are now routed to a reader replica (for clusters). This impacts all retrievers, the `Neo4jChunkReader` and `SinglePropertyExactMatchResolver` components.
+
+
+## 1.2.0
+
 ### Added
 - Made `relations` and `potential_schema` optional in `SchemaBuilder`.
 - Added a check to prevent the use of deprecated Cypher syntax for Neo4j versions 5.23.0 and above.
 - Added a `LexicalGraphBuilder` component to enable the import of the lexical graph (document, chunks) without performing entity and relation extraction.
+- Added a `Neo4jChunkReader` component to be able to read chunk text from the database.
 
 ### Changed
 - Vector and Hybrid retrievers used with `return_properties` now also return the node labels (`nodeLabels`) and the node's element ID (`id`).
 - `HybridRetriever` now filters out the embedding property index in `self.vector_index_name` from the retriever result by default.
 - Removed support for neo4j.AsyncDriver in the KG creation pipeline, affecting Neo4jWriter and related components.
 - Updated examples and unit tests to reflect the removal of async driver support.
 
+### Fixed
+- Resolved issue with `AzureOpenAIEmbeddings` incorrectly inheriting from `OpenAIEmbeddings`, now inherits from `BaseOpenAIEmbeddings`.
 
 ## 1.1.0
 

docs/source/api.rst

@@ -58,6 +58,14 @@ LexicalGraphBuilder
     :members:
     :exclude-members: component_inputs, component_outputs
 
+
+Neo4jChunkReader
+================
+
+.. autoclass:: neo4j_graphrag.experimental.components.neo4j_reader.Neo4jChunkReader
+    :members:
+    :exclude-members: component_inputs, component_outputs
+
 SchemaBuilder
 =============
 

docs/source/index.rst

@@ -295,10 +295,3 @@ Further information
 
 -   `The official Neo4j Python driver <https://github.com/neo4j/neo4j-python-driver>`_
 -   `Neo4j GenAI integrations <https://neo4j.com/docs/cypher-manual/current/genai-integrations/>`_
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`

docs/source/user_guide_kg_builder.rst

@@ -16,7 +16,7 @@ unstructured data.
 Pipeline Structure
 ******************
 
-A Knowledge Graph (KG) construction pipeline requires a few components:
+A Knowledge Graph (KG) construction pipeline requires a few components (some of the below components are optional):
 
 - **Document parser**: extract text from files (PDFs, ...).
 - **Document chunker**: split the text into smaller pieces of text, manageable by the LLM context window (token limit).
@@ -205,6 +205,47 @@ Example usage:
 See :ref:`kg-writer-section` to learn how to write the resulting nodes and relationships to Neo4j.
 
 
+Neo4j Chunk Reader
+==================
+
+The Neo4j chunk reader component is used to read text chunks from Neo4j. Text chunks can be created
+by the lexical graph builder or another process.
+
+.. code:: python
+
+    import neo4j
+    from neo4j_graphrag.experimental.components.neo4j_reader import Neo4jChunkReader
+    from neo4j_graphrag.experimental.components.types import LexicalGraphConfig
+
+    reader = Neo4jChunkReader(driver)
+    result = await reader.run()
+
+
+Configure node labels and relationship types
+---------------------------------------------
+
+Optionally, the document and chunk node labels can be configured using a `LexicalGraphConfig` object:
+
+.. code:: python
+
+    from neo4j_graphrag.experimental.components.neo4j_reader import Neo4jChunkReader
+    from neo4j_graphrag.experimental.components.types import LexicalGraphConfig, TextChunks
+
+    # optionally, define a LexicalGraphConfig object
+    # shown below with the default values
+    config = LexicalGraphConfig(
+        id_prefix="",  # used to prefix the chunk and document IDs
+        chunk_node_label="Chunk",
+        document_node_label="Document",
+        chunk_to_document_relationship_type="PART_OF_DOCUMENT",
+        next_chunk_relationship_type="NEXT_CHUNK",
+        node_to_chunk_relationship_type="PART_OF_CHUNK",
+        chunk_embedding_property="embeddings",
+    )
+    reader = Neo4jChunkReader(driver)
+    result = await reader.run(lexical_graph_config=config)
+
+
 Schema Builder
 ==============
 

examples/README.md

@@ -92,6 +92,8 @@ are listed in [the last section of this file](#customize).
 - [End to end example with explicit components and text input](./customize/build_graph/pipeline/kg_builder_from_text.py)
 - [End to end example with explicit components and PDF input](./customize/build_graph/pipeline/kg_builder_from_pdf.py)
 - [Process multiple documents](./customize/build_graph/pipeline/kg_builder_two_documents_entity_resolution.py)
+- [Export lexical graph creation into another pipeline](./customize/build_graph/pipeline/text_to_lexical_graph_to_entity_graph_two_pipelines.py)
+
 
 #### Components
 

examples/build_graph/simple_kg_builder_from_pdf.py

@@ -50,6 +50,7 @@ async def define_and_run_pipeline(
         entities=ENTITIES,
         relations=RELATIONS,
         potential_schema=POTENTIAL_SCHEMA,
+        neo4j_database=DATABASE,
     )
     return await kg_builder.run_async(file_path=str(file_path))
 
@@ -62,7 +63,7 @@ async def main() -> PipelineResult:
             "response_format": {"type": "json_object"},
         },
     )
-    with neo4j.GraphDatabase.driver(URI, auth=AUTH, database=DATABASE) as driver:
+    with neo4j.GraphDatabase.driver(URI, auth=AUTH) as driver:
         res = await define_and_run_pipeline(driver, llm)
     await llm.async_client.close()
     return res

examples/build_graph/simple_kg_builder_from_text.py

@@ -3,6 +3,8 @@
 
 This example assumes a Neo4j db is up and running. Update the credentials below
 if needed.
+
+NB: when building a KG from text, no 'Document' node is created in the Knowledge Graph.
 """
 
 import asyncio
@@ -11,6 +13,10 @@
 from neo4j_graphrag.embeddings import OpenAIEmbeddings
 from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline
 from neo4j_graphrag.experimental.pipeline.pipeline import PipelineResult
+from neo4j_graphrag.experimental.pipeline.types import (
+    EntityInputType,
+    RelationInputType,
+)
 from neo4j_graphrag.llm import LLMInterface
 from neo4j_graphrag.llm.openai_llm import OpenAILLM
 
@@ -21,12 +27,28 @@
 
 # Text to process
 TEXT = """The son of Duke Leto Atreides and the Lady Jessica, Paul is the heir of House Atreides,
-an aristocratic family that rules the planet Caladan."""
+an aristocratic family that rules the planet Caladan, the rainy planet, since 10191."""
 
 # Instantiate Entity and Relation objects. This defines the
 # entities and relations the LLM will be looking for in the text.
-ENTITIES = ["Person", "House", "Planet"]
-RELATIONS = ["PARENT_OF", "HEIR_OF", "RULES"]
+ENTITIES: list[EntityInputType] = [
+    # entities can be defined with a simple label...
+    "Person",
+    # ... or with a dict if more details are needed,
+    # such as a description:
+    {"label": "House", "description": "Family the person belongs to"},
+    # or a list of properties the LLM will try to attach to the entity:
+    {"label": "Planet", "properties": [{"name": "weather", "type": "STRING"}]},
+]
+# same thing for relationships:
+RELATIONS: list[RelationInputType] = [
+    "PARENT_OF",
+    {
+        "label": "HEIR_OF",
+        "description": "Used for inheritor relationship between father and sons",
+    },
+    {"label": "RULES", "properties": [{"name": "fromYear", "type": "INTEGER"}]},
+]
 POTENTIAL_SCHEMA = [
     ("Person", "PARENT_OF", "Person"),
     ("Person", "HEIR_OF", "House"),
@@ -47,6 +69,7 @@ async def define_and_run_pipeline(
         relations=RELATIONS,
         potential_schema=POTENTIAL_SCHEMA,
         from_pdf=False,
+        neo4j_database=DATABASE,
     )
     return await kg_builder.run_async(text=TEXT)
 
@@ -59,7 +82,7 @@ async def main() -> PipelineResult:
             "response_format": {"type": "json_object"},
         },
     )
-    with neo4j.GraphDatabase.driver(URI, auth=AUTH, database=DATABASE) as driver:
+    with neo4j.GraphDatabase.driver(URI, auth=AUTH) as driver:
         res = await define_and_run_pipeline(driver, llm)
     await llm.async_client.close()
     return res

examples/customize/answer/custom_prompt.py

@@ -23,7 +23,6 @@
 driver = neo4j.GraphDatabase.driver(
     URI,
     auth=AUTH,
-    database=DATABASE,
 )
 
 embedder = OpenAIEmbeddings()
@@ -33,6 +32,7 @@
     index_name=INDEX,
     retrieval_query="WITH node, score RETURN node.title as title, node.plot as plot",
     embedder=embedder,
+    neo4j_database=DATABASE,
 )
 
 llm = OpenAILLM(model_name="gpt-4o", model_params={"temperature": 0})

examples/customize/answer/langchain_compatiblity.py

@@ -21,7 +21,6 @@
 driver = neo4j.GraphDatabase.driver(
     URI,
     auth=AUTH,
-    database=DATABASE,
 )
 
 embedder = OpenAIEmbeddings(model="text-embedding-ada-002")
@@ -31,6 +30,7 @@
     index_name=INDEX,
     retrieval_query="WITH node, score RETURN node.title as title, node.plot as plot",
     embedder=embedder,  # type: ignore[arg-type, unused-ignore]
+    neo4j_database=DATABASE,
 )
 
 llm = ChatOpenAI(model="gpt-4o", temperature=0)

examples/customize/build_graph/components/chunk_reader/neo4j_chunk_reader.py

@@ -0,0 +1,21 @@
+import asyncio
+
+import neo4j
+from neo4j_graphrag.experimental.components.neo4j_reader import Neo4jChunkReader
+from neo4j_graphrag.experimental.components.types import LexicalGraphConfig, TextChunks
+
+
+async def main(driver: neo4j.Driver) -> TextChunks:
+    config = LexicalGraphConfig(  # only needed to overwrite the default values
+        chunk_node_label="TextPart",
+    )
+    reader = Neo4jChunkReader(driver)
+    result = await reader.run(lexical_graph_config=config)
+    return result
+
+
+if __name__ == "__main__":
+    with neo4j.GraphDatabase.driver(
+        "bolt://localhost:7687", auth=("neo4j", "password")
+    ) as driver:
+        print(asyncio.run(main(driver)))