-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Add scylladb module #8002
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Add scylladb module #8002
Changes from 4 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
86ed2cb
add scylladb module
mkorolyov bc7eb8b
Update docs/modules/databases/scylladb.md
mkorolyov 35ea1c2
remove com.datastax.cassandra:cassandra-driver-core from deps
mkorolyov 8590397
Merge branch 'main' into add-scylladb
mkorolyov f10e620
Merge remote-tracking branch 'origin/main' into add-scylladb
eddumelendez 1805c16
Polish
eddumelendez b5fa599
Add docs
eddumelendez af00b41
add custom config file. add ssl test and document it
mkorolyov 7badf3d
Fix format
eddumelendez 60f7dd4
remove temporary ssl test
mkorolyov ccd7e41
add ssl cqlsh test and docs
mkorolyov 8ac1c15
Merge branch 'main' into add-scylladb
mkorolyov 7a356f2
Merge branch 'main' into add-scylladb
mkorolyov 181472e
add withSsl method to ScyllaDBContainer
mkorolyov 19ae30a
add additional withSsl(certificate, keyfile)
mkorolyov 3b95f64
Merge branch 'main' into add-scylladb
mkorolyov 6ab9c7e
Polish
eddumelendez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,6 +47,7 @@ body: | |
| - QuestDB | ||
| - RabbitMQ | ||
| - Redpanda | ||
| - ScyllaDB | ||
| - Selenium | ||
| - Solace | ||
| - Solr | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,6 +47,7 @@ body: | |
| - QuestDB | ||
| - RabbitMQ | ||
| - Redpanda | ||
| - ScyllaDB | ||
| - Selenium | ||
| - Solace | ||
| - Solr | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,6 +47,7 @@ body: | |
| - Pulsar | ||
| - RabbitMQ | ||
| - Redpanda | ||
| - ScyllaDB | ||
| - Selenium | ||
| - Solace | ||
| - Solr | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # ScyllaDB Module | ||
|
|
||
| ## Usage example | ||
|
|
||
| This example connects to a ScyllaDB Cluster, creates a keyspace and asserts that is has been created. | ||
|
|
||
| <!--codeinclude--> | ||
| [Building CqlSession](../../../modules/scylladb/src/test/java/org/testcontainers/containers/ScyllaDBDriver4Test.java) inside_block:scylladb | ||
| <!--/codeinclude--> | ||
|
|
||
| ## Adding this module to your project dependencies | ||
|
|
||
| Add the following dependency to your `pom.xml`/`build.gradle` file: | ||
|
|
||
| === "Gradle" | ||
| ```groovy | ||
| testImplementation "org.testcontainers:scylladb:{{latest_version}}" | ||
| ``` | ||
|
|
||
| === "Maven" | ||
| ```xml | ||
| <dependency> | ||
| <groupId>org.testcontainers</groupId> | ||
| <artifactId>scylladb</artifactId> | ||
| <version>{{latest_version}}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| description = "Testcontainers :: ScyllaDB" | ||
|
|
||
| configurations.all { | ||
| resolutionStrategy { | ||
| force 'io.dropwizard.metrics:metrics-core:3.2.6' | ||
| } | ||
| } | ||
|
|
||
| dependencies { | ||
| api project(":database-commons") | ||
| api "com.scylladb:java-driver-core:4.15.0.0" | ||
|
|
||
| testImplementation 'org.assertj:assertj-core:3.24.2' | ||
| } |
194 changes: 194 additions & 0 deletions
194
modules/scylladb/src/main/java/org/testcontainers/containers/ScyllaDBContainer.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| package org.testcontainers.containers; | ||
|
|
||
| import com.github.dockerjava.api.command.InspectContainerResponse; | ||
| import org.apache.commons.io.IOUtils; | ||
| import org.testcontainers.containers.delegate.ScyllaDBDatabaseDelegate; | ||
| import org.testcontainers.delegate.DatabaseDelegate; | ||
| import org.testcontainers.ext.ScriptUtils; | ||
| import org.testcontainers.ext.ScriptUtils.ScriptLoadException; | ||
| import org.testcontainers.utility.DockerImageName; | ||
| import org.testcontainers.utility.MountableFile; | ||
|
|
||
| import java.io.IOException; | ||
| import java.net.InetSocketAddress; | ||
| import java.net.URL; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.Optional; | ||
|
|
||
| import javax.script.ScriptException; | ||
|
|
||
| /** | ||
| * Testcontainers implementation for ScyllaDB. | ||
| * <p> | ||
| * Supported image: {@code scylladb} | ||
| * <p> | ||
| * Exposed ports: 9042 | ||
| */ | ||
| public class ScyllaDBContainer<SELF extends ScyllaDBContainer<SELF>> extends GenericContainer<SELF> { | ||
|
|
||
| private static final DockerImageName DEFAULT_IMAGE_NAME = DockerImageName.parse("scylladb/scylla:5.2.9"); | ||
|
|
||
| public static final Integer CQL_PORT = 9042; | ||
|
|
||
| private static final String DEFAULT_LOCAL_DATACENTER = "datacenter1"; | ||
|
|
||
| private static final String CONTAINER_CONFIG_LOCATION = "/etc/scylla"; | ||
|
|
||
| private static final String USERNAME = "scylladb"; | ||
|
|
||
| private static final String PASSWORD = "scylladb"; | ||
|
|
||
| private String configLocation; | ||
|
|
||
| private String initScriptPath; | ||
|
|
||
| private boolean enableJmxReporting; | ||
|
|
||
| public ScyllaDBContainer(String dockerImageName) { | ||
| this(DockerImageName.parse(dockerImageName)); | ||
| } | ||
|
|
||
| public ScyllaDBContainer(DockerImageName dockerImageName) { | ||
| super(dockerImageName); | ||
| dockerImageName.assertCompatibleWith(DEFAULT_IMAGE_NAME); | ||
|
|
||
| addExposedPort(CQL_PORT); | ||
| this.enableJmxReporting = false; | ||
|
|
||
| withEnv("CASSANDRA_SNITCH", "GossipingPropertyFileSnitch"); | ||
| withEnv("JVM_OPTS", "-Dcassandra.skip_wait_for_gossip_to_settle=0 -Dcassandra.initial_token=0"); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. those are not relevant to scylla, you need commandline option for |
||
| withEnv("HEAP_NEWSIZE", "128M"); | ||
| withEnv("MAX_HEAP_SIZE", "1024M"); | ||
| withEnv("SCYLLADB_ENDPOINT_SNITCH", "GossipingPropertyFileSnitch"); | ||
| withEnv("SCYLLADB_DC", DEFAULT_LOCAL_DATACENTER); | ||
| } | ||
|
|
||
| @Override | ||
| protected void configure() { | ||
| optionallyMapResourceParameterAsVolume(CONTAINER_CONFIG_LOCATION, configLocation); | ||
| } | ||
|
|
||
| @Override | ||
| protected void containerIsStarted(InspectContainerResponse containerInfo) { | ||
| runInitScriptIfRequired(); | ||
| } | ||
|
|
||
| /** | ||
| * Load init script content and apply it to the database if initScriptPath is set | ||
| */ | ||
| private void runInitScriptIfRequired() { | ||
| if (initScriptPath != null) { | ||
| try { | ||
| URL resource = Thread.currentThread().getContextClassLoader().getResource(initScriptPath); | ||
| if (resource == null) { | ||
| logger().warn("Could not load classpath init script: {}", initScriptPath); | ||
| throw new ScriptLoadException( | ||
| "Could not load classpath init script: " + initScriptPath + ". Resource not found." | ||
| ); | ||
| } | ||
| String cql = IOUtils.toString(resource, StandardCharsets.UTF_8); | ||
| DatabaseDelegate databaseDelegate = new ScyllaDBDatabaseDelegate(this); | ||
| ScriptUtils.executeDatabaseScript(databaseDelegate, initScriptPath, cql); | ||
| } catch (IOException e) { | ||
| logger().warn("Could not load classpath init script: {}", initScriptPath); | ||
| throw new ScriptLoadException("Could not load classpath init script: " + initScriptPath, e); | ||
| } catch (ScriptException e) { | ||
| logger().error("Error while executing init script: {}", initScriptPath, e); | ||
| throw new ScriptUtils.UncategorizedScriptException( | ||
| "Error while executing init script: " + initScriptPath, | ||
| e | ||
| ); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Map (effectively replace) directory in Docker with the content of resourceLocation if resource location is not null | ||
| * | ||
| * Protected to allow for changing implementation by extending the class | ||
| * | ||
| * @param pathNameInContainer path in docker | ||
| * @param resourceLocation relative classpath to resource | ||
| */ | ||
| protected void optionallyMapResourceParameterAsVolume(String pathNameInContainer, String resourceLocation) { | ||
| Optional | ||
| .ofNullable(resourceLocation) | ||
| .map(MountableFile::forClasspathResource) | ||
| .ifPresent(mountableFile -> withCopyFileToContainer(mountableFile, pathNameInContainer)); | ||
| } | ||
|
|
||
| /** | ||
| * Initialize ScyllaDB with the custom overridden ScyllaDB configuration | ||
| * <p> | ||
| * Be aware, that Docker effectively replaces all /etc/sylladb content with the content of config location, so if | ||
| * scylladb.yaml in configLocation is absent or corrupted, then ScyllaDB just won't launch | ||
| * | ||
| * @param configLocation relative classpath with the directory that contains cassandra.yaml and other configuration files | ||
| */ | ||
| public SELF withConfigurationOverride(String configLocation) { | ||
| this.configLocation = configLocation; | ||
| return self(); | ||
| } | ||
|
|
||
| /** | ||
| * Initialize ScyllaDB with init CQL script | ||
| * <p> | ||
| * CQL script will be applied after container is started (see using WaitStrategy) | ||
| * | ||
| * @param initScriptPath relative classpath resource | ||
| */ | ||
| public SELF withInitScript(String initScriptPath) { | ||
| this.initScriptPath = initScriptPath; | ||
| return self(); | ||
| } | ||
|
|
||
| /** | ||
| * Initialize ScyllaDB client with JMX reporting enabled or disabled | ||
| */ | ||
| public SELF withJmxReporting(boolean enableJmxReporting) { | ||
| this.enableJmxReporting = enableJmxReporting; | ||
| return self(); | ||
| } | ||
|
|
||
| /** | ||
| * Get username | ||
| * | ||
| * By default ScyllaDB has authenticator: AllowAllAuthenticator in scylladb.yaml | ||
| * If username and password need to be used, then authenticator should be set as PasswordAuthenticator | ||
| * (through custom ScyllaDB configuration) and through CQL with default scylladb-scylladb credentials | ||
| * user management should be modified | ||
| */ | ||
| public String getUsername() { | ||
| return USERNAME; | ||
| } | ||
|
|
||
| /** | ||
| * Get password | ||
| * | ||
| * By default ScyllaDB has authenticator: AllowAllAuthenticator in scylladb.yaml | ||
| * If username and password need to be used, then authenticator should be set as PasswordAuthenticator | ||
| * (through custom Cassandra configuration) and through CQL with default scylladb-scylladb credentials | ||
| * user management should be modified | ||
| */ | ||
| public String getPassword() { | ||
| return PASSWORD; | ||
| } | ||
|
|
||
| /** | ||
| * Retrieve an {@link InetSocketAddress} for connecting to the ScyllaDB container via the driver. | ||
| * | ||
| * @return A InetSocketAddrss representation of this ScyllaDB container's host and port. | ||
| */ | ||
| public InetSocketAddress getContactPoint() { | ||
| return new InetSocketAddress(getHost(), getMappedPort(CQL_PORT)); | ||
| } | ||
|
|
||
| /** | ||
| * Retrieve the Local Datacenter for connecting to the ScyllaDB container via the driver. | ||
| * | ||
| * @return The configured local Datacenter name. | ||
| */ | ||
| public String getLocalDatacenter() { | ||
| return getEnvMap().getOrDefault("SCYLLADB_DC", DEFAULT_LOCAL_DATACENTER); | ||
| } | ||
| } | ||
69 changes: 69 additions & 0 deletions
69
...ylladb/src/main/java/org/testcontainers/containers/delegate/ScyllaDBDatabaseDelegate.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| package org.testcontainers.containers.delegate; | ||
|
|
||
| import com.datastax.oss.driver.api.core.CqlSession; | ||
| import com.datastax.oss.driver.api.core.DriverException; | ||
| import com.datastax.oss.driver.api.core.cql.ResultSet; | ||
| import org.slf4j.Logger; | ||
| import org.testcontainers.containers.ContainerState; | ||
| import org.testcontainers.containers.ScyllaDBContainer; | ||
| import org.testcontainers.delegate.AbstractDatabaseDelegate; | ||
| import org.testcontainers.exception.ConnectionCreationException; | ||
| import org.testcontainers.ext.ScriptUtils.ScriptStatementFailedException; | ||
| import org.testcontainers.utility.DockerLoggerFactory; | ||
|
|
||
| import java.net.InetSocketAddress; | ||
|
|
||
| public class ScyllaDBDatabaseDelegate extends AbstractDatabaseDelegate<CqlSession> { | ||
|
|
||
| public ScyllaDBDatabaseDelegate(ContainerState container) { | ||
| this.container = container; | ||
| } | ||
|
|
||
| protected Logger logger() { | ||
| return DockerLoggerFactory.getLogger(container.getCurrentContainerInfo().getName()); | ||
| } | ||
|
|
||
| private final ContainerState container; | ||
|
|
||
| @Override | ||
| protected CqlSession createNewConnection() { | ||
| try { | ||
| return CqlSession | ||
| .builder() | ||
| .addContactPoint( | ||
| new InetSocketAddress(container.getHost(), container.getMappedPort(ScyllaDBContainer.CQL_PORT)) | ||
| ) | ||
| .withLocalDatacenter("datacenter1") | ||
| .build(); | ||
| } catch (DriverException e) { | ||
| throw new ConnectionCreationException("Could not obtain cassandra connection", e); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public void execute( | ||
| String statement, | ||
| String scriptPath, | ||
| int lineNumber, | ||
| boolean continueOnError, | ||
| boolean ignoreFailedDrops | ||
| ) { | ||
| try { | ||
| ResultSet result = getConnection().execute(statement); | ||
| if (!result.wasApplied()) { | ||
| throw new ScriptStatementFailedException(statement, lineNumber, scriptPath); | ||
| } | ||
| } catch (DriverException e) { | ||
| throw new ScriptStatementFailedException(statement, lineNumber, scriptPath, e); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| protected void closeConnectionQuietly(CqlSession session) { | ||
| try { | ||
| session.close(); | ||
| } catch (Exception e) { | ||
| logger().error("Could not close cassandra connection", e); | ||
| } | ||
| } | ||
| } |
43 changes: 43 additions & 0 deletions
43
.../scylladb/src/main/java/org/testcontainers/containers/wait/ScyllaDBQueryWaitStrategy.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| package org.testcontainers.containers.wait; | ||
|
|
||
| import org.rnorth.ducttape.TimeoutException; | ||
| import org.testcontainers.containers.ContainerLaunchException; | ||
| import org.testcontainers.containers.delegate.ScyllaDBDatabaseDelegate; | ||
| import org.testcontainers.containers.wait.strategy.AbstractWaitStrategy; | ||
| import org.testcontainers.delegate.DatabaseDelegate; | ||
|
|
||
| import java.util.concurrent.TimeUnit; | ||
|
|
||
| import static org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess; | ||
|
|
||
| /** | ||
| * Waits until ScyllaDB returns its version | ||
| */ | ||
| public class ScyllaDBQueryWaitStrategy extends AbstractWaitStrategy { | ||
|
|
||
| private static final String SELECT_VERSION_QUERY = "SELECT release_version FROM system.local"; | ||
|
|
||
| private static final String TIMEOUT_ERROR = "Timed out waiting for ScyllaDB to be accessible for query execution"; | ||
|
|
||
| @Override | ||
| protected void waitUntilReady() { | ||
| // execute select version query until success or timeout | ||
| try { | ||
| retryUntilSuccess( | ||
| (int) startupTimeout.getSeconds(), | ||
| TimeUnit.SECONDS, | ||
| () -> { | ||
| getRateLimiter() | ||
| .doWhenReady(() -> { | ||
| try (DatabaseDelegate databaseDelegate = new ScyllaDBDatabaseDelegate(waitStrategyTarget)) { | ||
| databaseDelegate.execute(SELECT_VERSION_QUERY, "", 1, false, false); | ||
| } | ||
| }); | ||
| return true; | ||
| } | ||
| ); | ||
| } catch (TimeoutException e) { | ||
| throw new ContainerLaunchException(TIMEOUT_ERROR); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs to be checked, scylla docker doesn't use this variable