CDM-30 rename packages

Author: mieslep

README.md

@@ -24,15 +24,17 @@ tar -xvzf spark-3.3.1-bin-hadoop3.tgz
 
 # Steps for Data-Migration:
 
-1. `sparkConf.properties` file needs to be configured as applicable for the environment
+> :warning: Note that Version 4 of the tool is not backward-compatible with .properties files created in previous versions, and that package names have changed.
+
+1. `sparkConf.properties` file needs to be configured as applicable for the environment. Parameter descriptions and defaults are described in the file.
    > A sample Spark conf file configuration can be [found here](./src/resources/sparkConf.properties)
 2. Place the conf file where it can be accessed while running the job via spark-submit.
 3. Run the below job using `spark-submit` command as shown below:
 
 ```
 ./spark-submit --properties-file sparkConf.properties /
 --master "local[*]" /
---class datastax.astra.migrate.Migrate cassandra-data-migrator-3.x.x.jar &> logfile_name.txt
+--class datastax.cdm.job.Migrate cassandra-data-migrator-4.x.x.jar &> logfile_name.txt
 ```
 
 Note: 
@@ -41,26 +43,26 @@ Note:
 ```
 ./spark-submit --properties-file sparkConf.properties /
 --master "local[*]" --driver-memory 25G --executor-memory 25G /
---class datastax.astra.migrate.Migrate cassandra-data-migrator-3.x.x.jar &> logfile_name.txt
+--class datastax.cdm.job.Migrate cassandra-data-migrator-4.x.x.jar &> logfile_name.txt
 ```
 
 # Steps for Data-Validation:
 
-- To run the job in Data validation mode, use class option `--class datastax.astra.migrate.DiffData` as shown below
+- To run the job in Data validation mode, use class option `--class datastax.cdm.job.DiffData` as shown below
 
 ```
 ./spark-submit --properties-file sparkConf.properties /
 --master "local[*]" /
---class datastax.astra.migrate.DiffData cassandra-data-migrator-3.x.x.jar &> logfile_name.txt
+--class datastax.cdm.job.DiffData cassandra-data-migrator-4.x.x.jar &> logfile_name.txt
 ```
 
 - Validation job will report differences as “ERRORS” in the log file as shown below
 
 ```
-22/10/27 23:25:29 ERROR DiffJobSession: Missing target row found for key: Grapes %% 1 %% 2020-05-22 %% 2020-05-23T00:05:09.353Z %% skuid %% Aliquam faucibus
-22/10/27 23:25:29 ERROR DiffJobSession: Inserted missing row in target: Grapes %% 1 %% 2020-05-22 %% 2020-05-23T00:05:09.353Z %% skuid %% Aliquam faucibus
-22/10/27 23:25:30 ERROR DiffJobSession: Mismatch row found for key: Grapes %% 1 %% 2020-05-22 %% 2020-05-23T00:05:09.353Z %% skuid %% augue odio at quam Data:  (Index: 8 Origin: Hello 3 Target: Hello 2 )
-22/10/27 23:25:30 ERROR DiffJobSession: Updated mismatch row in target: Grapes %% 1 %% 2020-05-22 %% 2020-05-23T00:05:09.353Z %% skuid %% augue odio at quam
+23/04/06 08:43:06 ERROR DiffJobSession: Mismatch row found for key: [key3] Mismatch: Target Index: 1 Origin: valueC Target: value999) 
+23/04/06 08:43:06 ERROR DiffJobSession: Corrected mismatch row in target: [key3]
+23/04/06 08:43:06 ERROR DiffJobSession: Missing target row found for key: [key2]
+23/04/06 08:43:06 ERROR DiffJobSession: Inserted missing row in target: [key2]
 ```
 
 - Please grep for all `ERROR` from the output log files to get the list of missing and mismatched records.
@@ -70,18 +72,18 @@ Note:
   - Update any mismatched records between origin and target (makes target same as origin). 
 - Enable/disable this feature using one or both of the below setting in the config file
 ```
-spark.target.autocorrect.missing                    true|false
-spark.target.autocorrect.mismatch                   true|false
+spark.cdm.autocorrect.missing                     false|true
+spark.cdm.autocorrect.mismatch                    false|true
 ```
 Note:
 - The validation job will never delete records from target i.e. it only adds or updates data on target
 
 # Migrating specific partition ranges
-- You can also use the tool to migrate specific partition ranges using class option `--class datastax.astra.migrate.MigratePartitionsFromFile` as shown below
+- You can also use the tool to migrate specific partition ranges using class option `--class datastax.cdm.job.MigratePartitionsFromFile` as shown below
 ```
 ./spark-submit --properties-file sparkConf.properties /
 --master "local[*]" /
---class datastax.astra.migrate.MigratePartitionsFromFile cassandra-data-migrator-3.x.x.jar &> logfile_name.txt
+--class datastax.cdm.job.MigratePartitionsFromFile cassandra-data-migrator-4.x.x.jar &> logfile_name.txt
 ```
 
 When running in above mode the tool assumes a `partitions.csv` file to be present in the current folder in the below format, where each line (`min,max`) represents a partition-range 
@@ -99,6 +101,7 @@ This mode is specifically useful to processes a subset of partition-ranges that
 - Supports migration/validation of advanced DataTypes ([Sets](https://docs.datastax.com/en/dse/6.8/cql/cql/cql_reference/refDataTypes.html#refDataTypes__set), [Lists](https://docs.datastax.com/en/dse/6.8/cql/cql/cql_reference/refDataTypes.html#refDataTypes__list), [Maps](https://docs.datastax.com/en/dse/6.8/cql/cql/cql_reference/refDataTypes.html#refDataTypes__map), [UDTs](https://docs.datastax.com/en/dse/6.8/cql/cql/cql_reference/refDataTypes.html#refDataTypes__udt))
 - Filter records from `Origin` using `writetimes` and/or CQL conditions and/or min/max token-range
 - Supports adding `constants` as new columns on `Target`
+- Supports expanding `Map` columns on `Origin` into multiple records on `Target`
 - Fully containerized (Docker and K8s friendly)
 - SSL Support (including custom cipher algorithms)
 - Migrate from any Cassandra `Origin` ([Apache Cassandra®](https://cassandra.apache.org) / [DataStax Enterprise™](https://www.datastax.com/products/datastax-enterprise) / [DataStax Astra DB™](https://www.datastax.com/products/datastax-astra)) to any Cassandra `Target` ([Apache Cassandra®](https://cassandra.apache.org) / [DataStax Enterprise™](https://www.datastax.com/products/datastax-enterprise) / [DataStax Astra DB™](https://www.datastax.com/products/datastax-astra))
@@ -110,7 +113,7 @@ This mode is specifically useful to processes a subset of partition-ranges that
 1. Clone this repo
 2. Move to the repo folder `cd cassandra-data-migrator`
 3. Run the build `mvn clean package` (Needs Maven 3.8.x)
-4. The fat jar (`cassandra-data-migrator-3.x.x.jar`) file should now be present in the `target` folder
+4. The fat jar (`cassandra-data-migrator-4.x.x.jar`) file should now be present in the `target` folder
 
 # Contributors
 Checkout all our wonderful contributors [here](./CONTRIBUTING.md#contributors).

SIT/features/01_constant_column/cdm.txt

@@ -1,3 +1,3 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
-fixData datastax.astra.migrate.DiffData fix.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties
+fixData datastax.cdm.job.DiffData fix.properties

SIT/features/02_explode_map/cdm.txt

@@ -1,3 +1,3 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
-fixData datastax.astra.migrate.DiffData fix.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties
+fixData datastax.cdm.job.DiffData fix.properties

SIT/features/03_explode_map_with_constants/cdm.txt

@@ -1,3 +1,3 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
-fixData datastax.astra.migrate.DiffData fix.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties
+fixData datastax.cdm.job.DiffData fix.properties

SIT/smoke/01_basic_kvp/cdm.txt

@@ -1,2 +1,2 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties

SIT/smoke/02_autocorrect_kvp/cdm.txt

@@ -1,2 +1,2 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties

SIT/smoke/03_ttl_writetime/cdm.txt

@@ -1,3 +1,3 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
-fixData datastax.astra.migrate.DiffData fix.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties
+fixData datastax.cdm.job.DiffData fix.properties

SIT/smoke/04_counters/cdm.txt

@@ -1,4 +1,4 @@
-migrateData datastax.astra.migrate.Migrate migrate.properties
-validateData datastax.astra.migrate.DiffData migrate.properties
-fixData datastax.astra.migrate.DiffData fix.properties
-fixForce datastax.astra.migrate.DiffData fix-force.properties
+migrateData datastax.cdm.job.Migrate migrate.properties
+validateData datastax.cdm.job.DiffData migrate.properties
+fixData datastax.cdm.job.DiffData fix.properties
+fixForce datastax.cdm.job.DiffData fix-force.properties

pom.xml

@@ -1,7 +1,7 @@
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
 
-  <groupId>datastax.astra.migrate</groupId>
+  <groupId>datastax.cdm</groupId>
   <artifactId>cassandra-data-migrator</artifactId>
   <version>${revision}</version>
   <packaging>jar</packaging>

src/main/java/datastax/astra/migrate/cql/CqlHelper.java renamed to src/main/java/datastax/cdm/cql/CqlHelper.java

@@ -1,20 +1,22 @@
-package datastax.astra.migrate.cql;
+package datastax.cdm.cql;
 
 import com.datastax.oss.driver.api.core.ConsistencyLevel;
 import com.datastax.oss.driver.api.core.CqlSession;
 import com.datastax.oss.driver.api.core.cql.Row;
-import datastax.astra.migrate.MigrateDataType;
-import datastax.astra.migrate.cql.features.*;
-import datastax.astra.migrate.properties.KnownProperties;
-import datastax.astra.migrate.properties.PropertyHelper;
+import datastax.cdm.job.MigrateDataType;
+import datastax.cdm.data.PKFactory;
+import datastax.cdm.feature.Feature;
+import datastax.cdm.feature.FeatureFactory;
+import datastax.cdm.feature.Featureset;
+import datastax.cdm.cql.statement.*;
+import datastax.cdm.properties.KnownProperties;
+import datastax.cdm.properties.PropertyHelper;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.util.*;
 
-import datastax.astra.migrate.cql.statements.*;
-
 public class CqlHelper {
     private final Logger logger = LoggerFactory.getLogger(this.getClass().getName());