Skip to content

Commit b43bd39

Browse files
mieslepmieslep
authored
Merge pull request #117 from datastax/issue/CDM-23
Issue/cdm 23
2 parents 402ee11 + 127bc4e commit b43bd39

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+2589
-744
lines changed

‎SIT/features/02_explode_map/breakData.cql

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
DELETE FROM target.feature_explode_map WHERE key='key2';
3+
UPDATE target.feature_explode_map SET value='value999' WHERE key='key3' AND fruit='apples';
4+
UPDATE target.feature_explode_map SET fruit_qty=999 WHERE key='key3' AND fruit='oranges';
5+
DELETE FROM target.feature_explode_map WHERE key='key3' AND fruit='kiwi';
6+
SELECT * FROM target.feature_explode_map;
7+

‎SIT/features/02_explode_map/cdm.fixData.assert

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Read Record Count: 3
2+
Mismatch Record Count: 2
3+
Corrected Mismatch Record Count: 2
4+
Missing Record Count: 5
5+
Corrected Missing Record Count: 5
6+
Valid Record Count: 5
7+
Skipped Record Count: 0

‎SIT/features/02_explode_map/cdm.migrateData.assert

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Read Record Count: 3
2+
Skipped Record Count: 0
3+
Write Record Count: 12
4+
Error Record Count: 0

‎SIT/features/02_explode_map/cdm.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
migrateData datastax.astra.migrate.Migrate migrate.properties
2+
validateData datastax.astra.migrate.DiffData migrate.properties
3+
fixData datastax.astra.migrate.DiffData fix.properties

‎SIT/features/02_explode_map/cdm.validateData.assert

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Read Record Count: 3
2+
Mismatch Record Count: 0
3+
Corrected Mismatch Record Count: 0
4+
Missing Record Count: 0
5+
Corrected Missing Record Count: 0
6+
Valid Record Count: 12
7+
Skipped Record Count: 0

‎SIT/features/02_explode_map/execute.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash -e
2+
3+
workingDir="$1"
4+
cd "$workingDir"
5+
6+
/local/cdm.sh -f cdm.txt -s migrateData -d "$workingDir" > cdm.migrateData.out 2>cdm.migrateData.err
7+
/local/cdm-assert.sh -f cdm.migrateData.out -a cdm.migrateData.assert -d "$workingDir"
8+
9+
/local/cdm.sh -f cdm.txt -s validateData -d "$workingDir" > cdm.validateData.out 2>cdm.validateData.err
10+
/local/cdm-assert.sh -f cdm.validateData.out -a cdm.validateData.assert -d "$workingDir"
11+
12+
cqlsh -u $CASS_USERNAME -p $CASS_PASSWORD $CASS_CLUSTER -f $workingDir/breakData.cql > $workingDir/breakData.out 2> $workingDir/breakData.err
13+
14+
/local/cdm.sh -f cdm.txt -s fixData -d "$workingDir" > cdm.fixData.out 2>cdm.fixData.err
15+
/local/cdm-assert.sh -f cdm.fixData.out -a cdm.fixData.assert -d "$workingDir"

‎SIT/features/02_explode_map/expected.cql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT * FROM target.feature_explode_map;

‎SIT/features/02_explode_map/expected.out

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
2+
key | fruit | fruit_qty | value
3+
------+---------+-----------+--------
4+
key1 | apples | 3 | valueA
5+
key1 | bananas | 2 | valueA
6+
key1 | grapes | 11 | valueA
7+
key1 | oranges | 5 | valueA
8+
key3 | apples | 5 | valueC
9+
key3 | bananas | 4 | valueC
10+
key3 | kiwi | 42 | valueC
11+
key3 | oranges | 7 | valueC
12+
key2 | apples | 4 | valueB
13+
key2 | bananas | 3 | valueB
14+
key2 | oranges | 6 | valueB
15+
key2 | pears | 7 | valueB
16+
17+
(12 rows)

‎SIT/features/02_explode_map/fix.properties

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
spark.origin.host cdm-sit-cass
2+
spark.origin.port 9042
3+
spark.origin.username cassandra
4+
spark.origin.password cassandra
5+
spark.origin.keyspaceTable source.feature_explode_map
6+
7+
spark.target.host cdm-sit-cass
8+
spark.target.port 9042
9+
spark.target.username cassandra
10+
spark.target.password cassandra
11+
spark.target.keyspaceTable target.feature_explode_map
12+
13+
spark.numSplits 1
14+
15+
spark.query.origin key,value,fruits
16+
spark.query.origin.partitionKey key
17+
spark.query.target.id key,fruit
18+
spark.query.types 0,0,5%0%1
19+
20+
spark.cdm.cql.feature.explodeMap.origin.name fruits
21+
spark.cdm.cql.feature.explodeMap.target.name.key fruit
22+
spark.cdm.cql.feature.explodeMap.target.name.value fruit_qty
23+
24+
spark.target.autocorrect.missing true
25+
spark.target.autocorrect.mismatch true

‎SIT/features/02_explode_map/migrate.properties

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
spark.origin.host cdm-sit-cass
2+
spark.origin.port 9042
3+
spark.origin.username cassandra
4+
spark.origin.password cassandra
5+
spark.origin.keyspaceTable source.feature_explode_map
6+
7+
spark.target.host cdm-sit-cass
8+
spark.target.port 9042
9+
spark.target.username cassandra
10+
spark.target.password cassandra
11+
spark.target.keyspaceTable target.feature_explode_map
12+
13+
spark.numSplits 1
14+
15+
spark.query.origin key,value,fruits
16+
spark.query.origin.partitionKey key
17+
spark.query.target.id key,fruit
18+
spark.query.types 0,0,5%0%1
19+
20+
spark.cdm.cql.feature.explodeMap.origin.name fruits
21+
spark.cdm.cql.feature.explodeMap.target.name.key fruit
22+
spark.cdm.cql.feature.explodeMap.target.name.value fruit_qty

0 commit comments

Comments
 (0)