Skip to content

Commit 1c3486a

Browse files
committed
Minor updates to documentation for clarity
1 parent 6e7c4a2 commit 1c3486a

File tree

2 files changed

+13
-5
lines changed

2 files changed

+13
-5
lines changed

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,14 @@ When running in above mode the tool assumes a `partitions.csv` file to be presen
9797
```
9898
This mode is specifically useful to processes a subset of partition-ranges that may have failed during a previous run.
9999

100+
> **Note:**
101+
> Here is a quick tip to prepare `partitions.csv` from the log file,
102+
103+
```
104+
grep "ERROR CopyJobSession: Error with PartitionRange" /path/to/logfile_name.txt | awk '{print $13","$15}' > partitions.csv
105+
```
106+
107+
100108
# Perform large-field Guardrail violation checks
101109
- The tool can be used to identify large fields from a table that may break you cluster guardrails (e.g. AstraDB has a 10MB limit for a single large field) `--class datastax.astra.migrate.Guardrail` as shown below
102110
```

src/resources/runCommands.txt

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,16 @@ curl -OL https://downloads.datastax.com/enterprise/cqlsh-astra.tar.gz
44
wget https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz
55

66
// Migrate
7-
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar
8-
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar &> table_out.log
7+
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar &> log_name_$(date +%Y%m%d_%H_%M).log
8+
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar &> table_out_$(date +%Y%m%d_%H_%M).log
99
// If target keyspace oand/or table name is different than origin, then add --conf spark.target.keyspaceTable="keyspace2.table2"
1010
// Add option --verbose for verbose output
1111

1212
// Random Partitioner Run Command
13-
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --conf spark.origin.minPartition=-1 --conf spark.origin.maxPartition=170141183460469231731687303715884105728 --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar
13+
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --conf spark.origin.minPartition=-1 --conf spark.origin.maxPartition=170141183460469231731687303715884105728 --class datastax.astra.migrate.Migrate /<path>/cassandra-data-migrator-3.4.*.jar &> log_name_$(date +%Y%m%d_%H_%M).log
1414

1515
// Validate
16-
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.DiffData /<path>/cassandra-data-migrator-3.4.*.jar &> table_out.log
16+
spark-submit --properties-file /<path>/cdm.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.DiffData /<path>/cassandra-data-migrator-3.4.*.jar &> table_out_$(date +%Y%m%d_%H_%M).log
1717

1818
// Guardrail check (identify large fields)
19-
spark-submit --properties-file /<path>/cdmGuardrail.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.Guardrail /<path>/cassandra-data-migrator-3.4.*.jar &> table_out.log
19+
spark-submit --properties-file /<path>/cdmGuardrail.properties --conf spark.origin.keyspaceTable="keyspace.table" --master "local[*]" --driver-memory 25G --executor-memory 25G --class datastax.astra.migrate.Guardrail /<path>/cassandra-data-migrator-3.4.*.jar &> table_out_$(date +%Y%m%d_%H_%M).log

0 commit comments

Comments
 (0)