Skip to content

Commit ed5fe2c

Browse files
mieslepmieslep
committed
CDM-55 adding guardrail feature
1 parent c086174 commit ed5fe2c

27 files changed

+668
-263
lines changed

‎SIT/features/05_guardrail/cdm.fixData.assert

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Read Record Count: 4
2+
Mismatch Record Count: 0
3+
Corrected Mismatch Record Count: 0
4+
Missing Record Count: 0
5+
Corrected Missing Record Count: 0
6+
Valid Record Count: 10
7+
Skipped Record Count: 6

‎SIT/features/05_guardrail/cdm.guardrailCheck.assert

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Read Record Count: 4
2+
Valid Record Count: 10
3+
Large Record Count: 6

‎SIT/features/05_guardrail/cdm.migrateData.assert

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Read Record Count: 4
2+
Skipped Record Count: 6
3+
Write Record Count: 10
4+
Error Record Count: 0

‎SIT/features/05_guardrail/cdm.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
migrateData com.datastax.cdm.job.Migrate migrate.properties
2+
guardrailCheck com.datastax.cdm.job.GuardrailCheck migrate.properties
3+
validateData com.datastax.cdm.job.DiffData migrate.properties
4+
fixData com.datastax.cdm.job.DiffData fix.properties

‎SIT/features/05_guardrail/cdm.validateData.assert

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Read Record Count: 4
2+
Mismatch Record Count: 0
3+
Corrected Mismatch Record Count: 0
4+
Missing Record Count: 0
5+
Corrected Missing Record Count: 0
6+
Valid Record Count: 10
7+
Skipped Record Count: 6

‎SIT/features/05_guardrail/execute.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash -e
2+
3+
workingDir="$1"
4+
cd "$workingDir"
5+
6+
/local/cdm.sh -f cdm.txt -s guardrailCheck -d "$workingDir" > cdm.guardrailCheck.out 2>cdm.guardrailCheck.err
7+
/local/cdm-assert.sh -f cdm.guardrailCheck.out -a cdm.guardrailCheck.assert -d "$workingDir"
8+
9+
/local/cdm.sh -f cdm.txt -s migrateData -d "$workingDir" > cdm.migrateData.out 2>cdm.migrateData.err
10+
/local/cdm-assert.sh -f cdm.migrateData.out -a cdm.migrateData.assert -d "$workingDir"
11+
12+
/local/cdm.sh -f cdm.txt -s validateData -d "$workingDir" > cdm.validateData.out 2>cdm.validateData.err
13+
/local/cdm-assert.sh -f cdm.validateData.out -a cdm.validateData.assert -d "$workingDir"
14+
15+
/local/cdm.sh -f cdm.txt -s fixData -d "$workingDir" > cdm.fixData.out 2>cdm.fixData.err
16+
/local/cdm-assert.sh -f cdm.fixData.out -a cdm.fixData.assert -d "$workingDir"

‎SIT/features/05_guardrail/expected.cql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT * FROM target.feature_guardrail;

‎SIT/features/05_guardrail/expected.out

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
2+
key | fruit | fruit_taste | value
3+
-------------+---------+-------------+--------
4+
badMapValue | bananas | squishy | valueA
5+
badMapValue | grapes | sour | valueA
6+
badMapValue | oranges | sweet | valueA
7+
clean | apples | delicious | valueA
8+
clean | bananas | squishy | valueA
9+
clean | grapes | sour | valueA
10+
clean | oranges | sweet | valueA
11+
badMapKey | bananas | squishy | valueA
12+
badMapKey | grapes | sour | valueA
13+
badMapKey | oranges | sweet | valueA
14+
15+
(10 rows)

‎SIT/features/05_guardrail/fix.properties

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
spark.cdm.origin.connect.host cdm-sit-cass
2+
spark.cdm.target.connect.host cdm-sit-cass
3+
4+
spark.cdm.schema.origin.keyspaceTable origin.feature_guardrail
5+
spark.cdm.schema.target.keyspaceTable target.feature_guardrail
6+
spark.cdm.perfops.numParts 1
7+
8+
spark.cdm.feature.explodeMap.origin.name fruits
9+
spark.cdm.feature.explodeMap.target.name.key fruit
10+
spark.cdm.feature.explodeMap.target.name.value fruit_taste
11+
12+
spark.cdm.feature.guardrail.colSizeInKB 1
13+
14+
spark.cdm.autocorrect.missing true
15+
spark.cdm.autocorrect.mismatch true

‎SIT/features/05_guardrail/migrate.properties

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
spark.cdm.origin.connect.host cdm-sit-cass
2+
spark.cdm.target.connect.host cdm-sit-cass
3+
4+
spark.cdm.schema.origin.keyspaceTable origin.feature_guardrail
5+
spark.cdm.schema.target.keyspaceTable target.feature_guardrail
6+
spark.cdm.perfops.numParts 1
7+
8+
spark.cdm.feature.explodeMap.origin.name fruits
9+
spark.cdm.feature.explodeMap.target.name.key fruit
10+
spark.cdm.feature.explodeMap.target.name.value fruit_taste
11+
12+
spark.cdm.feature.guardrail.colSizeInKB 1

0 commit comments

Comments
 (0)