Skip to content

Commit b73e2f2

Browse files
michael-simonsmichael-simons
committed
GH-2245 - Provide additional scripts to run tests with retries against clusters.
We must ensure that SDN 6 runs successfully against cluster deployments. Our bookmark code is already under tests (explicitly via the JUnit 5 Neo4j cluster extension) under load. However, we should run the whole test suite, too. This change does the following - Provide additional selectors that can exclude tests that made no sense in a cluster environment - Seed all tests with the bookmarks from the fixtures - Capture the bookmarks after the test and assert with them as starting point - Provide additional Ci scripts to run them in a controlled manner, providing retries to react on cluster failures during tests, much like application developers would do. This closes #2245.
1 parent 65e3dc6 commit b73e2f2

File tree

75 files changed

+2668
-1542
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+2668
-1542
lines changed

‎ci/runClusterTests.sh

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env bash
2+
3+
# This script is used to run SDN6's integration test against a Neo4j cluster via `runClusterTests.java`. The idea behind it is:
4+
# 1. Build a local deployment of SDN 6 including a test jar containing all tests
5+
# 2. Extract the version, direct dependencies and the repo path from the Maven build descriptor and replace the placeholders
6+
# in the template.
7+
# 3. Execute the Java script file via JBang.
8+
#
9+
# The result code of this script will be 0 in a successful run, a non-zero value otherwise. The Java program will try
10+
# upto 100 times to get a completely successful test run, retrying on error cases that might happen in a cluster.
11+
#
12+
# Run this script with a pair of environmental values to point it to a cluster:
13+
# SDN_NEO4J_URL=neo4j+s://your.neo4j.cluster.io SDN_NEO4J_PASSWORD=yourPassword ./ci/runClusterTests.sh
14+
15+
set -euo pipefail
16+
17+
CI_BASEDIR=$(dirname "$0")
18+
BASEDIR=$(realpath $CI_BASEDIR/..)
19+
CLUSTER_TEST_DIR=$BASEDIR/target/cluster-tests
20+
21+
(
22+
cd $BASEDIR
23+
SDN_VERSION=$(./mvnw help:evaluate -Dexpression=project.version -q -DforceStdout)
24+
25+
mkdir -p $CLUSTER_TEST_DIR
26+
27+
# Create the distribution and deploy it into the target folder itself
28+
./mvnw -Pgenerate-test-jar -DskipTests clean deploy -DaltDeploymentRepository=snapshot-repo::default::file:///$CLUSTER_TEST_DIR/snapshot-repo
29+
30+
# Massage the directory name into something sed is happy with
31+
SNAPSHOT_REPO=$(printf '%s\n' "$CLUSTER_TEST_DIR/snapshot-repo" | sed -e 's/[\/&]/\\&/g')
32+
33+
# Create a plain list of dependencies
34+
./mvnw dependency:list -DexcludeTransitive | sed -n -e 's/^\[INFO\] //p' > $CLUSTER_TEST_DIR/dependencies.txt
35+
36+
# Update repository path, version and dependencies in template
37+
sed -e s/\$SDN_VERSION/$SDN_VERSION/ -e s/\$SNAPSHOT_REPO/$SNAPSHOT_REPO/ $CI_BASEDIR/runClusterTests.template.java |\
38+
awk -F: -v deps=$CLUSTER_TEST_DIR/dependencies.txt -v target=$CLUSTER_TEST_DIR/runClusterTests.java '
39+
/\/\/\$ADDITIONAL_DEPENDENCIES/ {
40+
while((getline < deps) > 0) {
41+
print "//DEPS " $1 ":" $2 ":" $4 > target
42+
}
43+
next
44+
}
45+
{print > target}'
46+
47+
# clean up
48+
rm $CLUSTER_TEST_DIR/dependencies.txt
49+
50+
# Prepare run
51+
chmod +x $CLUSTER_TEST_DIR/runClusterTests.java && cp src/test/resources/logback-silent.xml $CLUSTER_TEST_DIR/logback.xml
52+
)
53+
54+
jbang $CLUSTER_TEST_DIR/runClusterTests.java

‎ci/runClusterTests.template.java

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
///usr/bin/env jbang "$0" "$@" ; exit $?
2+
//JAVA 11+
3+
//REPOS mavencentral,spring-libs-snapshot=https://repo.spring.io/libs-snapshot,snapshot-repo=file://$SNAPSHOT_REPO
4+
5+
// This is a template to create a Java script runnable via JBang https://github.com/jbangdev/jbang and Java 11 to run.
6+
// It takes most of the integration tests and runs them against Neo4j. It will react on exceptions happening due to cluster
7+
// changes and retry them accordingly. While you can point it against a non-cluster deployment, it wouldn't make much sense,
8+
// as this scenario is already covered based on the standard testing.
9+
// The file is named `template.java` as it has some placeholders that needs to be replaced by the orchestrating `runClusterTests.sh`.
10+
11+
//FILES logback.xml
12+
13+
//DEPS org.junit.platform:junit-platform-launcher:1.7.1
14+
//DEPS org.springframework.data:spring-data-neo4j:$SDN_VERSION
15+
//DEPS org.springframework.data:spring-data-neo4j:$SDN_VERSION:tests@test-jar
16+
//$ADDITIONAL_DEPENDENCIES
17+
18+
import static java.util.stream.Collectors.partitioningBy;
19+
import static java.util.stream.Collectors.toList;
20+
import static org.junit.platform.engine.discovery.ClassNameFilter.includeClassNamePatterns;
21+
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectPackage;
22+
import static org.junit.platform.launcher.TagFilter.excludeTags;
23+
24+
import java.io.IOException;
25+
import java.nio.file.Files;
26+
import java.nio.file.StandardCopyOption;
27+
import java.util.List;
28+
29+
import org.junit.platform.engine.DiscoverySelector;
30+
import org.junit.platform.engine.discovery.DiscoverySelectors;
31+
import org.junit.platform.launcher.TestExecutionListener;
32+
import org.junit.platform.launcher.TestIdentifier;
33+
import org.junit.platform.launcher.core.LauncherConfig;
34+
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
35+
import org.junit.platform.launcher.core.LauncherFactory;
36+
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
37+
import org.neo4j.driver.internal.retry.ExponentialBackoffRetryLogic;
38+
import org.slf4j.LoggerFactory;
39+
import org.springframework.data.neo4j.core.support.RetryExceptionPredicate;
40+
41+
public class runClusterTests {
42+
43+
public static void main(String... args) throws IOException {
44+
45+
var logConfig = Files.createTempFile("logback", ".xml");
46+
try (var s = runClusterTests.class.getResourceAsStream("logback.xml")) {
47+
Files.copy(s, logConfig, StandardCopyOption.REPLACE_EXISTING);
48+
}
49+
System.setProperty("logback.configurationFile", logConfig.toAbsolutePath().normalize().toString());
50+
51+
var log = LoggerFactory.getLogger(runClusterTests.class);
52+
53+
var listener = new SummaryGeneratingListener();
54+
var launcher = LauncherFactory
55+
.create(LauncherConfig.builder()
56+
.addTestExecutionListeners(new TestExecutionListener() {
57+
@Override public void executionStarted(TestIdentifier testIdentifier) {
58+
if (testIdentifier.isContainer() && testIdentifier.getParentId().isPresent()) {
59+
log.debug(testIdentifier.getUniqueId());
60+
}
61+
}
62+
})
63+
.addTestExecutionListeners(listener).build());
64+
65+
var canRetry = new RetryExceptionPredicate().or(ExponentialBackoffRetryLogic::isRetryable);
66+
67+
var selectors = List.<DiscoverySelector>of(selectPackage("org.springframework.data.neo4j.integration"));
68+
var maxRetries = 100;
69+
var counter = 0;
70+
71+
while (!selectors.isEmpty() && ++counter <= maxRetries) {
72+
log.info("Attempt {}/{}", counter, maxRetries);
73+
var request = LauncherDiscoveryRequestBuilder.request()
74+
.selectors(selectors)
75+
.filters(
76+
includeClassNamePatterns(".*IT.*"),
77+
excludeTags("incompatible-with-clusters")
78+
)
79+
.build();
80+
81+
launcher.execute(request);
82+
83+
var failures = listener.getSummary().getFailures();
84+
85+
if (failures.isEmpty()) {
86+
log.info("Finished succesfully after {} attempts.", counter);
87+
System.exit(0);
88+
}
89+
90+
var failuresByState = failures.stream().collect(partitioningBy(failure -> {
91+
var ex = new Throwable[] { failure.getException() };
92+
if (ex[0] instanceof AssertionError) {
93+
ex = ((AssertionError) ex[0]).getSuppressed();
94+
}
95+
for (var candidate : ex) {
96+
do {
97+
if (canRetry.test(candidate)) {
98+
return true;
99+
}
100+
candidate = candidate.getCause();
101+
} while (candidate != null);
102+
}
103+
return false;
104+
}));
105+
106+
if (!failuresByState.get(false).isEmpty()) {
107+
log.error("The following tests failed in non retryable ways:");
108+
failuresByState.get(false).forEach(failure -> {
109+
log.error(failure.getTestIdentifier().getUniqueId());
110+
failure.getException().printStackTrace();
111+
});
112+
System.exit(1);
113+
}
114+
115+
selectors = failuresByState.get(true).stream()
116+
.peek(failure -> log.info("{} will be retried", failure.getTestIdentifier().getUniqueId()))
117+
.map(failure -> DiscoverySelectors.selectUniqueId(failure.getTestIdentifier().getUniqueId()))
118+
.collect(toList());
119+
}
120+
log.error("Several tests failed despite a number of retries.");
121+
System.exit(1);
122+
}
123+
}

‎pom.xml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -741,6 +741,25 @@
741741
</plugins>
742742
</build>
743743
</profile>
744+
745+
<profile>
746+
<id>generate-test-jar</id>
747+
<build>
748+
<plugins>
749+
<plugin>
750+
<groupId>org.apache.maven.plugins</groupId>
751+
<artifactId>maven-jar-plugin</artifactId>
752+
<executions>
753+
<execution>
754+
<goals>
755+
<goal>test-jar</goal>
756+
</goals>
757+
</execution>
758+
</executions>
759+
</plugin>
760+
</plugins>
761+
</build>
762+
</profile>
744763
</profiles>
745764

746765
</project>

‎src/test/java/org/springframework/data/neo4j/documentation/repositories/custom_queries/CustomQueriesIT.java

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,18 @@
3030
import org.springframework.context.annotation.Bean;
3131
import org.springframework.context.annotation.Configuration;
3232
import org.springframework.data.neo4j.config.AbstractNeo4jConfig;
33+
import org.springframework.data.neo4j.core.DatabaseSelectionProvider;
34+
import org.springframework.data.neo4j.core.transaction.Neo4jBookmarkManager;
35+
import org.springframework.data.neo4j.core.transaction.Neo4jTransactionManager;
3336
import org.springframework.data.neo4j.documentation.domain.MovieEntity;
3437
import org.springframework.data.neo4j.documentation.domain.PersonEntity;
3538
import org.springframework.data.neo4j.integration.movies.shared.CypherUtils;
3639
import org.springframework.data.neo4j.repository.Neo4jRepository;
3740
import org.springframework.data.neo4j.repository.config.EnableNeo4jRepositories;
41+
import org.springframework.data.neo4j.test.BookmarkCapture;
3842
import org.springframework.data.neo4j.test.Neo4jExtension;
3943
import org.springframework.data.neo4j.test.Neo4jIntegrationTest;
44+
import org.springframework.transaction.PlatformTransactionManager;
4045
import org.springframework.transaction.annotation.EnableTransactionManagement;
4146

4247
/**
@@ -72,12 +77,13 @@ void customRepositoryFragmentsShouldWork(
7277
// end::custom-queries-test[]
7378

7479
@BeforeAll
75-
static void setupData(@Autowired Driver driver) throws IOException {
80+
static void setupData(@Autowired Driver driver, @Autowired BookmarkCapture bookmarkCapture) throws IOException {
7681

77-
try (Session session = driver.session()) {
82+
try (Session session = driver.session(bookmarkCapture.createSessionConfig())) {
7883
session.run("MATCH (n) DETACH DELETE n").consume();
7984
session.run("MATCH (n) DETACH DELETE n").consume();
8085
CypherUtils.loadCypherFromResource("/data/movies.cypher", session);
86+
bookmarkCapture.seedWith(session.lastBookmark());
8187
}
8288
}
8389

@@ -99,5 +105,17 @@ public Driver driver() {
99105
protected Collection<String> getMappingBasePackages() {
100106
return Collections.singletonList(MovieEntity.class.getPackage().getName());
101107
}
108+
109+
@Bean
110+
public BookmarkCapture bookmarkCapture() {
111+
return new BookmarkCapture();
112+
}
113+
114+
@Override
115+
public PlatformTransactionManager transactionManager(Driver driver, DatabaseSelectionProvider databaseNameProvider) {
116+
117+
BookmarkCapture bookmarkCapture = bookmarkCapture();
118+
return new Neo4jTransactionManager(driver, databaseNameProvider, Neo4jBookmarkManager.create(bookmarkCapture));
119+
}
102120
}
103121
}

‎src/test/java/org/springframework/data/neo4j/integration/conversion_imperative/CustomTypesIT.java

Lines changed: 31 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,6 @@
3131
import org.neo4j.driver.Driver;
3232
import org.neo4j.driver.Result;
3333
import org.neo4j.driver.Session;
34-
import org.neo4j.driver.SessionConfig;
3534
import org.neo4j.driver.TransactionWork;
3635
import org.neo4j.driver.Values;
3736
import org.neo4j.driver.summary.ResultSummary;
@@ -40,16 +39,21 @@
4039
import org.springframework.context.annotation.Configuration;
4140
import org.springframework.core.convert.converter.GenericConverter;
4241
import org.springframework.data.neo4j.config.AbstractNeo4jConfig;
42+
import org.springframework.data.neo4j.core.DatabaseSelectionProvider;
4343
import org.springframework.data.neo4j.core.Neo4jOperations;
4444
import org.springframework.data.neo4j.core.convert.Neo4jConversions;
45+
import org.springframework.data.neo4j.core.transaction.Neo4jBookmarkManager;
46+
import org.springframework.data.neo4j.core.transaction.Neo4jTransactionManager;
4547
import org.springframework.data.neo4j.integration.shared.conversion.PersonWithCustomId;
4648
import org.springframework.data.neo4j.integration.shared.conversion.ThingWithCustomTypes;
4749
import org.springframework.data.neo4j.repository.Neo4jRepository;
4850
import org.springframework.data.neo4j.repository.config.EnableNeo4jRepositories;
4951
import org.springframework.data.neo4j.repository.query.Query;
52+
import org.springframework.data.neo4j.test.BookmarkCapture;
5053
import org.springframework.data.neo4j.test.Neo4jExtension.Neo4jConnectionSupport;
5154
import org.springframework.data.neo4j.test.Neo4jIntegrationTest;
5255
import org.springframework.data.repository.query.Param;
56+
import org.springframework.transaction.PlatformTransactionManager;
5357
import org.springframework.transaction.annotation.EnableTransactionManagement;
5458

5559
/**
@@ -66,14 +70,13 @@ public class CustomTypesIT {
6670

6771
private final Neo4jOperations neo4jOperations;
6872

73+
private final BookmarkCapture bookmarkCapture;
74+
6975
@Autowired
70-
public CustomTypesIT(Driver driver, Neo4jOperations neo4jOperations) {
76+
public CustomTypesIT(Driver driver, Neo4jOperations neo4jOperations, BookmarkCapture bookmarkCapture) {
7177
this.driver = driver;
7278
this.neo4jOperations = neo4jOperations;
73-
}
74-
75-
SessionConfig getSessionConfig() {
76-
return SessionConfig.defaultConfig();
79+
this.bookmarkCapture = bookmarkCapture;
7780
}
7881

7982
TransactionWork<ResultSummary> createPersonWithCustomId(PersonWithCustomId.PersonId assignedId) {
@@ -84,29 +87,32 @@ TransactionWork<ResultSummary> createPersonWithCustomId(PersonWithCustomId.Perso
8487

8588
@BeforeEach
8689
void setupData() {
87-
try (Session session = driver.session()) {
90+
try (Session session = driver.session(bookmarkCapture.createSessionConfig())) {
8891
session.writeTransaction(transaction -> {
8992
transaction.run("MATCH (n) detach delete n").consume();
9093
transaction.run("CREATE (:CustomTypes{customType:'XYZ'})").consume();
9194
return null;
9295
});
96+
bookmarkCapture.seedWith(session.lastBookmark());
9397
}
9498
}
9599

96100
@Test
97101
void deleteByCustomId() {
98102

99103
PersonWithCustomId.PersonId id = new PersonWithCustomId.PersonId(customIdValueGenerator.incrementAndGet());
100-
try (Session session = driver.session(getSessionConfig())) {
104+
try (Session session = driver.session(bookmarkCapture.createSessionConfig())) {
101105
session.writeTransaction(createPersonWithCustomId(id));
106+
bookmarkCapture.seedWith(session.lastBookmark());
102107
}
103108

104109
assertThat(neo4jOperations.count(PersonWithCustomId.class)).isEqualTo(1L);
105110
neo4jOperations.deleteById(id, PersonWithCustomId.class);
106111

107-
try (Session session = driver.session(getSessionConfig())) {
112+
try (Session session = driver.session(bookmarkCapture.createSessionConfig())) {
108113
Result result = session.run("MATCH (p:PersonWithCustomId) return count(p) as count");
109114
assertThat(result.single().get("count").asLong()).isEqualTo(0);
115+
bookmarkCapture.seedWith(session.lastBookmark());
110116
}
111117
}
112118

@@ -118,17 +124,19 @@ void deleteAllByCustomId() {
118124
.limit(2)
119125
.collect(Collectors.toList());
120126
try (
121-
Session session = driver.session(getSessionConfig());
127+
Session session = driver.session(bookmarkCapture.createSessionConfig());
122128
) {
123129
ids.forEach(id -> session.writeTransaction(createPersonWithCustomId(id)));
130+
bookmarkCapture.seedWith(session.lastBookmark());
124131
}
125132

126133
assertThat(neo4jOperations.count(PersonWithCustomId.class)).isEqualTo(2L);
127134
neo4jOperations.deleteAllById(ids, PersonWithCustomId.class);
128135

129-
try (Session session = driver.session(getSessionConfig())) {
136+
try (Session session = driver.session(bookmarkCapture.createSessionConfig())) {
130137
Result result = session.run("MATCH (p:PersonWithCustomId) return count(p) as count");
131138
assertThat(result.single().get("count").asLong()).isEqualTo(0);
139+
bookmarkCapture.seedWith(session.lastBookmark());
132140
}
133141
}
134142

@@ -210,5 +218,17 @@ public Neo4jConversions neo4jConversions() {
210218
protected Collection<String> getMappingBasePackages() {
211219
return Collections.singletonList(ThingWithCustomTypes.class.getPackage().getName());
212220
}
221+
222+
@Bean
223+
public BookmarkCapture bookmarkCapture() {
224+
return new BookmarkCapture();
225+
}
226+
227+
@Override
228+
public PlatformTransactionManager transactionManager(Driver driver, DatabaseSelectionProvider databaseNameProvider) {
229+
230+
BookmarkCapture bookmarkCapture = bookmarkCapture();
231+
return new Neo4jTransactionManager(driver, databaseNameProvider, Neo4jBookmarkManager.create(bookmarkCapture));
232+
}
213233
}
214234
}

0 commit comments

Comments
 (0)