Property graph index in-memory #13939
-
|
Hello! I tried the property graph index soon after it was released. One issue I had at that time was the reliance on Neo4j. What I am looking for is a property graph index that can be stored in-memory like the Is this currently possible with the property graph index? I have not looked into the nitty gritty details of the implementation. Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 6 replies
-
|
Yes, it is currently possible to use the property graph index in a manner that allows it to be stored in-memory and also in an external graph database like Neo4j. The Here is an example: from llama_index.core.indices.property_graph import PropertyGraphIndex
from llama_index.graph_stores.neo4j import Neo4jPropertyGraphStore
# Create a Neo4jPropertyGraphStore instance
graph_store = Neo4jPropertyGraphStore(
username="neo4j",
password="neo4j",
url="bolt://localhost:7687",
database="neo4j"
)
# Create the index
index = PropertyGraphIndex.from_documents(
documents,
property_graph_store=graph_store,
)This example shows how to instantiate a Additionally, you can set up Neo4j locally using Docker: docker run \
-p 7474:7474 -p 7687:7687 \
-v $PWD/data:/data -v $PWD/plugins:/plugins \
--name neo4j-apoc \
-e NEO4J_apoc_export_file_enabled=true \
-e NEO4J_apoc_import_file_enabled=true \
-e NEO4J_apoc_import_file_use__neo4j__config=true \
-e NEO4JLABS_PLUGINS=\[\"apoc\"\] \
neo4j:latestAfter launching Neo4j, you can access it at http://localhost:7474/ with the default username/password of |
Beta Was this translation helpful? Give feedback.
-
|
@dosu, thanks for your response but your example only shows how to use Neo4j as a property graph store. Is there something, for the property graph index, like the |
Beta Was this translation helpful? Give feedback.
-
|
Yes, there is an equivalent in-memory Here is the relevant code snippet: class SimplePropertyGraphStore(PropertyGraphStore):
"""Simple Labelled Property Graph Store.
This class implements a simple in-memory labelled property graph store.
Args:
graph (Optional[LabelledPropertyGraph]): Labelled property graph to initialize the store.
"""
supports_structured_queries: bool = False
supports_vector_queries: bool = False
def __init__(
self,
graph: Optional[LabelledPropertyGraph] = None,
) -> None:
self.graph = graph or LabelledPropertyGraph()
def get(
self,
properties: Optional[dict] = None,
ids: Optional[List[str]] = None,
) -> List[LabelledNode]:
"""Get nodes."""
nodes = list(self.graph.nodes.values())
if properties:
nodes = [
n
for n in nodes
if any(n.properties.get(k) == v for k, v in properties.items())
]
# Filter by node_ids
if ids:
nodes = [n for n in nodes if n.id in ids]
return nodes
def get_triplets(
self,
entity_names: Optional[List[str]] = None,
relation_names: Optional[List[str]] = None,
properties: Optional[dict] = None,
ids: Optional[List[str]] = None,
) -> List[Triplet]:
"""Get triplets."""
# if nothing is passed, return empty list
if not ids and not properties and not entity_names and not relation_names:
return []
triplets = self.graph.get_triplets()
if entity_names:
triplets = [
t
for t in triplets
if t[0].id in entity_names or t[2].id in entity_names
]
if relation_names:
triplets = [t for t in triplets if t[1].id in relation_names]
if properties:
triplets = [
t
for t in triplets
if any(
t[0].properties.get(k) == v
or t[1].properties.get(k) == v
or t[2].properties.get(k) == v
for k, v in properties.items()
)
]
# Filter by node_ids
if ids:
triplets = [
t for t in triplets if any(t[0].id == i or t[2].id == i for i in ids)
]
return triplets
def get_rel_map(
self,
graph_nodes: List[LabelledNode],
depth: int = 2,
limit: int = 30,
ignore_rels: Optional[List[str]] = None,
) -> List[Triplet]:
"""Get depth-aware rel map."""
triplets = []
cur_depth = 0
graph_triplets = self.get_triplets(ids=[gn.id for gn in graph_nodes])
seen_triplets = set()
while len(graph_triplets) > 0 and cur_depth < depth:
triplets.extend(graph_triplets)
# get next depth
graph_triplets = self.get_triplets(
entity_names=[t[2].id for t in graph_triplets]
)
graph_triplets = [t for t in graph_triplets if str(t) not in seen_triplets]
seen_triplets.update([str(t) for t in graph_triplets])
cur_depth += 1
ignore_rels = ignore_rels or []
triplets = [t for t in triplets if t[1].id not in ignore_rels]
return triplets[:limit]
def upsert_nodes(self, nodes: List[LabelledNode]) -> None:
"""Add nodes."""
for node in nodes:
self.graph.add_node(node)
def upsert_relations(self, relations: List[Relation]) -> None:
"""Add relations."""
for relation in relations:
self.graph.add_relation(relation)This class provides methods to get nodes, get triplets, upsert nodes, and upsert relations, among other functionalities [1]. |
Beta Was this translation helpful? Give feedback.
-
|
Perfect, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
This is not exactly on topic with the original question but it seems that the |
Beta Was this translation helpful? Give feedback.
-
|
The error I am getting is elusive. It appears (but I cannot confirm yet) that the retrieval from the property graph fails on certain relations. These relations are not always consistent because the relations may differ (do they?) every time I create a new index even if it is from the same data. This error does not happen if the |
Beta Was this translation helpful? Give feedback.
-
|
To resolve the issue with the Here is the relevant part of the def get_triplets(
self,
entity_names: Optional[List[str]] = None,
relation_names: Optional[List[str]] = None,
properties: Optional[dict] = None,
ids: Optional[List[str]] = None,
) -> List[Triplet]:
"""Get triplets."""
# if nothing is passed, return empty list
if not ids and not properties and not entity_names and not relation_names:
return []
triplets = self.graph.get_triplets()
if entity_names:
triplets = [
t
for t in triplets
if t[0].id in entity_names or t[2].id in entity_names
]
if relation_names:
triplets = [t for t in triplets if t[1].id in relation_names]
if properties:
triplets = [
t
for t in triplets
if any(
t[0].properties.get(k) == v
or t[1].properties.get(k) == v
or t[2].properties.get(k) == v
for k, v in properties.items()
)
]
# Filter by node_ids
if ids:
triplets = [
t for t in triplets if any(t[0].id == i or t[2].id == i for i in ids)
]
return tripletsEnsure that the For example, the def test_get() -> None:
g = SimplePropertyGraphStore()
e1 = EntityNode(name="e1")
e2 = EntityNode(name="e2", properties={"key": "value"})
r = Relation(label="r", source_id=e1.id, target_id=e2.id)
g.upsert_nodes([e1, e2])
g.upsert_relations([r])
assert g.get_triplets() == []
assert g.get_triplets(entity_names=["e1"]) == [(e1, r, e2)]
assert g.get_triplets(entity_names=["e2"]) == [(e1, r, e2)]
assert g.get_triplets(relation_names=["r"]) == [(e1, r, e2)]
assert g.get_triplets(properties={"key": "value"}) == [(e1, r, e2)]By ensuring your implementation aligns with these test cases, you can better diagnose and resolve the inconsistencies in relation retrieval. |
Beta Was this translation helpful? Give feedback.
-
|
@dosu Finally, I managed to print the full trace of the exception using where Given the error, I believe the fix is along the following lines, at the time of construction of the property graph index: graph_index = PropertyGraphIndex(
...
embed_kg_nodes=True if not global__neo4j_disable else False,
...
)This does work! |
Beta Was this translation helpful? Give feedback.
-
|
The By setting graph_index = PropertyGraphIndex(
...
embed_kg_nodes=True if not global__neo4j_disable else False,
...
)This approach ensures that the nodes are embedded when not using Neo4j, which aligns with the default behavior of the Additionally, ensure that the filtering logic in the |
Beta Was this translation helpful? Give feedback.
Yes, there is an equivalent in-memory
graph_storefor the property graph index that you can use instead of Neo4j. TheSimplePropertyGraphStoreclass in the LlamaIndex framework implements a simple in-memory labelled property graph store.Here is the relevant code snippet: