GH-2203 - Support list of list of relationships/nodes mapping.

Author: meistermeier

src/main/asciidoc/appendix/custom-queries.adoc

@@ -49,6 +49,82 @@ To get the right object(s) back, it is required to _collect_ the relationships a
 
 With this result as a single record it is possible for Spring Data Neo4j to add all related nodes correctly to the root node.
 
+[[custom-queries.for-relationships.long-paths]]
+=== Reaching deeper into the graph
+
+The example above assumes that you are only trying to fetch the first level of related nodes.
+This is sometimes not enough and there are maybe nodes deeper in the graph that should also be part of the mapped instance.
+There are two ways to achieve this: Database-side or client-side reduction.
+
+For this the example from above should also contain `Movies` on the `Persons` that get returned with the initial `Movie`.
+
+.Example for 'The Matrix' and 'Keanu Reeves'
+image::movie-graph-deep.png[]
+
+[[custom-queries.for-relationships.long-paths.database]]
+==== Database-side reduction
+
+Keeping in mind that Spring Data Neo4j can only properly process record based, the result for one entity instance needs to be in one record.
+Using https://neo4j.com/docs/cypher-manual/current/syntax/patterns/#cypher-pattern-path-variables[Cypher's path] capabilities is a valid option to fetch all branches in the graph.
+
+[source,cypher]
+.Naive path-based approach
+----
+MATCH p=(m:Movie{title: 'The Matrix'})<-[:ACTED_IN]-(:Person)-[:ACTED_IN*..0]->(:Movie)
+RETURN p;
+----
+
+This will result in multiple paths that are not merged within one record.
+It is possible to call `collect(p)` but Spring Data Neo4j does not understand the concept of paths in the mapping process.
+Thus, nodes and relationships needs to get extracted for the result.
+
+[source,cypher]
+.Extracting nodes and relationships
+----
+MATCH p=(m:Movie{title: 'The Matrix'})<-[:ACTED_IN]-(:Person)-[:ACTED_IN*..0]->(:Movie)
+RETURN m, nodes(p), relationships(p);
+----
+
+Because there are multiple paths that lead from 'The Matrix' to another movie, the result still won't be a single record.
+This is where https://neo4j.com/docs/cypher-manual/current/functions/list/#functions-reduce[Cypher's reduce function] comes into play.
+
+[source,cypher]
+.Reducing nodes and relationships
+----
+MATCH p=(m:Movie{title: 'The Matrix'})<-[:ACTED_IN]-(:Person)-[:ACTED_IN*..0]->(:Movie)
+WITH collect(p) as paths, m
+WITH m,
+reduce(a=[], node in reduce(b=[], c in [aa in paths | nodes(aa)] | b + c) | case when node in a then a else a + node end) as nodes,
+reduce(d=[], relationship in reduce(e=[], f in [dd in paths | relationships(dd)] | e + f) | case when relationship in d then d else d + relationship end) as relationships
+RETURN m, relationships, nodes;
+----
+
+The `reduce` function allows us to flatten the nodes and relationships from various paths.
+As a result we will get a tuple similar to <<custom-queries.for-relationships.one.record>> but with a mixture of relationship types or nodes in the collections.
+
+[[custom-queries.for-relationships.long-paths.client]]
+==== Client-side reduction
+
+If the reduction should happen on the client-side, Spring Data Neo4j enables you to map also lists of lists of relationships or nodes.
+Still, the requirement applies that the returned record should contain all information to hydrate the resulting entity instance correctly.
+
+[source,cypher]
+.Collect nodes and relationships from path
+----
+MATCH p=(m:Movie{title: 'The Matrix'})<-[:ACTED_IN]-(:Person)-[:ACTED_IN*..0]->(:Movie)
+RETURN m, collect(nodes(p)), collect(relationships(p));
+----
+
+The additional `collect` statement creates lists in the format:
+----
+[[rel1, rel2], [rel3, rel4]]
+----
+Those lists will now get converted during the mapping process into a flat list.
+
+NOTE: Deciding if you want to go with client-side or database-side reduction depends on the amount of data that will get generated.
+All the paths needs to get created in the database's memory first when the `reduce` function is used.
+On the other hand a large amount of data that needs to get merged on the client-side results in a higher memory usage there.
+
 [[custom-queries.parameters]]
 == Parameters in custom queries
 

src/main/asciidoc/appendix/query-creation.adoc

@@ -88,9 +88,8 @@ The above return part will then look like:
 The map projection and pattern comprehension used by SDN ensures that only the properties and relationships you have defined are getting queried.
 
 In cases where you have self-referencing nodes or creating schemas that potentially lead to cycles in the data that gets returned,
-SDN falls back to a path-based query creation that does a wildcard mapping on all available relationship types reachable from
-the initial domain entity.
-
-The return pattern looks similar to this:
-
-`RETURN n{..., __paths__: [p = (n)-[:HAS|KNOWS*]-() | p]}`
+SDN falls back to a cascading / data-driven query creation.
+Starting with an initial query that looks for the specific node and considering the conditions,
+it steps through the resulting nodes and, if their relationships are also mapped, would create further queries on the fly.
+This query creation and execution loop will continue until no query finds new relationships or nodes.
+The way of the creation can be seen analogue to the save/update process.

src/main/asciidoc/img/movie-graph-deep.png

@@ -445,16 +445,17 @@ private Optional<Object> createInstanceOfRelationships(Neo4jPersistentProperty p
 			Collection<Relationship> allMatchingTypeRelationshipsInResult = new ArrayList<>();
 			Collection<Node> allNodesWithMatchingLabelInResult = new ArrayList<>();
 
-			// Grab everything else
+			// Retrieve all relationships from the result's list(s)
 			StreamSupport.stream(allValues.values().spliterator(), false)
 					.filter(MappingSupport.isListContainingOnly(listType, this.relationshipType))
-					.flatMap(entry -> entry.asList(Value::asRelationship).stream())
+					.flatMap(entry -> MappingSupport.extractRelationships(listType, entry).stream())
 					.filter(r -> r.type().equals(typeOfRelationship) || relationshipDescription.isDynamic())
 					.forEach(allMatchingTypeRelationshipsInResult::add);
 
+			// Retrieve all nodes from the result's list(s)
 			StreamSupport.stream(allValues.values().spliterator(), false)
 					.filter(MappingSupport.isListContainingOnly(listType, this.nodeType))
-					.flatMap(entry -> entry.asList(Value::asNode).stream())
+					.flatMap(entry -> MappingSupport.extractNodes(listType, entry).stream())
 					.filter(n -> n.hasLabel(targetLabel)).collect(Collectors.toList())
 					.forEach(allNodesWithMatchingLabelInResult::add);
 

src/main/java/org/springframework/data/neo4j/core/mapping/DefaultNeo4jEntityConverter.java

@@ -16,15 +16,20 @@
 package org.springframework.data.neo4j.core.mapping;
 
 import java.util.AbstractMap.SimpleEntry;
+import java.util.ArrayList;
 import java.util.Collection;
 import java.util.Collections;
+import java.util.HashSet;
 import java.util.Map;
 import java.util.Objects;
+import java.util.function.Function;
 import java.util.function.Predicate;
 import java.util.stream.Collectors;
 
 import org.apiguardian.api.API;
 import org.neo4j.driver.Value;
+import org.neo4j.driver.types.Node;
+import org.neo4j.driver.types.Relationship;
 import org.neo4j.driver.types.Type;
 import org.springframework.data.mapping.PersistentPropertyAccessor;
 import org.springframework.lang.Nullable;
@@ -80,8 +85,17 @@ public static Collection<?> unifyRelationshipValue(Neo4jPersistentProperty prope
 	public static Predicate<Value> isListContainingOnly(Type collectionType, Type requiredType) {
 
 		Predicate<Value> containsOnlyRequiredType = entry -> {
+			// either this is a list containing other list of possible the same required type
+			// or the type exists directly in the list
 			for (Value listEntry : entry.values()) {
-				if (!listEntry.hasType(requiredType)) {
+				if (listEntry.hasType(collectionType)) {
+					boolean listInListCorrectType = true;
+					for (Value listInListEntry : entry.asList(Function.identity())) {
+						listInListCorrectType = listInListCorrectType && isListContainingOnly(collectionType, requiredType)
+								.test(listInListEntry);
+					}
+					return listInListCorrectType;
+				} else if (!listEntry.hasType(requiredType)) {
 					return false;
 				}
 			}
@@ -92,6 +106,40 @@ public static Predicate<Value> isListContainingOnly(Type collectionType, Type re
 		return isList.and(containsOnlyRequiredType);
 	}
 
+	static Collection<Relationship> extractRelationships(Type collectionType, Value entry) {
+
+		Collection<Relationship> relationships = new HashSet<>();
+
+		for (Value listEntry : entry.values()) {
+			if (listEntry.hasType(collectionType)) {
+				for (Value listInListEntry : entry.asList(Function.identity())) {
+					relationships.addAll(extractRelationships(collectionType, listInListEntry));
+				}
+			} else {
+				relationships.add(listEntry.asRelationship());
+			}
+		}
+		return relationships;
+	}
+
+	static Collection<Node> extractNodes(Type collectionType, Value entry) {
+
+		// There can be multiple relationships leading to the same node.
+		// Thus we need a collection implementation that supports duplicates.
+		Collection<Node> nodes = new ArrayList<>();
+
+		for (Value listEntry : entry.values()) {
+			if (listEntry.hasType(collectionType)) {
+				for (Value listInListEntry : entry.asList(Function.identity())) {
+					nodes.addAll(extractNodes(collectionType, listInListEntry));
+				}
+			} else {
+				nodes.add(listEntry.asNode());
+			}
+		}
+		return nodes;
+	}
+
 	private MappingSupport() {}
 
 	/**

src/main/java/org/springframework/data/neo4j/core/mapping/MappingSupport.java

@@ -1107,6 +1107,46 @@ void findEntityWithRelationshipViaQuery(@Autowired RelationshipRepository reposi
 
 		}
 
+		@Test
+		void findEntityWithRelationshipViaPathQuery(@Autowired RelationshipRepository repository) {
+
+			long personId;
+			long hobbyNodeId;
+			long petNode1Id;
+			long petNode2Id;
+
+			try (Session session = createSession()) {
+				Record record = session
+						.run("CREATE (n:PersonWithRelationship{name:'Freddie'})-[:Has]->(h1:Hobby{name:'Music'}), "
+								+ "(n)-[:Has]->(p1:Pet{name: 'Jerry'}), (n)-[:Has]->(p2:Pet{name: 'Tom'}) " + "RETURN n, h1, p1, p2")
+						.single();
+
+				Node personNode = record.get("n").asNode();
+				Node hobbyNode1 = record.get("h1").asNode();
+				Node petNode1 = record.get("p1").asNode();
+				Node petNode2 = record.get("p2").asNode();
+
+				personId = personNode.id();
+				hobbyNodeId = hobbyNode1.id();
+				petNode1Id = petNode1.id();
+				petNode2Id = petNode2.id();
+			}
+
+			PersonWithRelationship loadedPerson = repository.getPersonWithRelationshipsViaPathQuery();
+			assertThat(loadedPerson.getName()).isEqualTo("Freddie");
+			assertThat(loadedPerson.getId()).isEqualTo(personId);
+			Hobby hobby = loadedPerson.getHobbies();
+			assertThat(hobby).isNotNull();
+			assertThat(hobby.getId()).isEqualTo(hobbyNodeId);
+			assertThat(hobby.getName()).isEqualTo("Music");
+
+			List<Pet> pets = loadedPerson.getPets();
+			Pet comparisonPet1 = new Pet(petNode1Id, "Jerry");
+			Pet comparisonPet2 = new Pet(petNode2Id, "Tom");
+			assertThat(pets).containsExactlyInAnyOrder(comparisonPet1, comparisonPet2);
+
+		}
+
 		@Test
 		void findEntityWithRelationshipWithAssignedId(@Autowired PetRepository repository) {
 
@@ -3985,6 +4025,10 @@ interface RelationshipRepository extends Neo4jRepository<PersonWithRelationship,
 				+ "return n, petRels, pets, collect(r2) as hobbyRels, collect(h) as hobbies")
 		PersonWithRelationship getPersonWithRelationshipsViaQuery();
 
+		@Query("MATCH p=(n:PersonWithRelationship{name:'Freddie'})-[:Has*]->(something) "
+				+ "return n, collect(relationships(p)), collect(nodes(p))")
+		PersonWithRelationship getPersonWithRelationshipsViaPathQuery();
+
 		PersonWithRelationship findByPetsName(String petName);
 
 		PersonWithRelationship findByName(String name);