Skip to content

Commit ad75e41

Browse files
mfmaher2mfmaher2
committed
Merge branch 'main' of github.com:datastax/cassandra-data-migrator
Adding CI to main branch
2 parents 0c9618d + f3e012a commit ad75e41

29 files changed

+871
-4
lines changed

‎.github/pull_request_template.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<!-- Thanks for sending a pull request! -->
2+
3+
**What this PR does**:
4+
5+
**Which issue(s) this PR fixes**:
6+
Fixes #<issue number>
7+
8+
**Checklist:**
9+
- [ ] Automated Tests added/updated
10+
- [ ] Documentation added/updated
11+
- [ ] CLA Signed: [DataStax CLA](https://cla.datastax.com/)

‎CONTRIBUTING.md

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,26 @@ get started are described here.
3232
address a non-trivial issue that has been reported, you can follow the steps that the original user
3333
reported. Updating issues with such findings is immensely helpful to the maintainers.
3434

35+
### Commits
36+
37+
Keep your changes **focused**. Each commit should have a single, clear purpose expressed in its
38+
message.
39+
40+
Resist the urge to "fix" cosmetic issues (add/remove blank lines, move methods, etc.) in existing
41+
code. This adds cognitive load for reviewers, who have to figure out which changes are relevant to
42+
the actual issue. If you see legitimate issues, like typos, address them in a separate commit (it's
43+
fine to group multiple typo fixes in a single commit).
44+
45+
Commit message subjects start with a capital letter, use the imperative form and do **not** end
46+
with a period.
47+
48+
Avoid catch-all messages like "Minor cleanup", "Various fixes", etc. They don't provide any useful
49+
information to reviewers, and might be a sign that your commit contains unrelated changes.
50+
51+
We don't enforce a particular subject line length limit, but try to keep it short.
52+
53+
You can add more details after the subject line, separated by a blank line.
54+
3555
### Pull Requests
3656

3757
All contributions to this project are made in the form of pull requests, including:
@@ -55,6 +75,9 @@ All contributions to this project are made in the form of pull requests, includi
5575
and requires some study of the codebase. If you take the time to understand the code a bit before submitting
5676
pull requests, then the maintainers will take the time to help you make your pull requests better.
5777

78+
- Like commits, pull requests should be focused on a single, clearly stated goal.
79+
Contributors need to sign the [DataStax CLA](https://cla.datastax.com/).
80+
5881
### Maintainers
5982

6083
We are looking for more community ownership in this project. That means that we will be supportive of
@@ -133,4 +156,4 @@ For recognizing contributions, please follow [this documentation](https://allcon
133156

134157
<!-- ALL-CONTRIBUTORS-LIST:END -->
135158

136-
---
159+
---

‎README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
[![License Apache2](https://img.shields.io/hexpm/l/plug.svg)](http://www.apache.org/licenses/LICENSE-2.0)
12
[![Star on Github](https://img.shields.io/github/stars/datastax/cassandra-data-migrator.svg?style=social)](https://github.com/datastax/cassandra-data-migrator/stargazers)
23

34
# cassandra-data-migrator
@@ -114,4 +115,4 @@ This mode is specifically useful to processes a subset of partition-ranges that
114115
# Contributors
115116
Checkout all our wonderful contributors [here](./CONTRIBUTING.md#contributors).
116117

117-
---
118+
---

‎SIT/.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
setup.out
2+
setup.err
3+
execute.out
4+
execute.err
5+
actual.out
6+
actual.err

‎SIT/Makefile

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
2+
all: setup test_smoke env_teardown
3+
4+
setup: build env_setup
5+
6+
reset: build env_reset
7+
reset_hard: build env_teardown env_setup
8+
9+
build:
10+
cd .. && mvn clean package
11+
12+
test_smoke: reset test_smoke_cmd
13+
test_smoke_cmd:
14+
./test.sh -p smoke
15+
16+
env_setup:
17+
./environment.sh -m setup -j ../target/cassandra-data-migrator*.jar
18+
env_reset:
19+
./environment.sh -m reset -j ../target/cassandra-data-migrator*.jar
20+
env_validate:
21+
./environment.sh -m validate
22+
env_teardown:
23+
./environment.sh -m teardown
24+

‎SIT/cdm.sh

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/bin/bash
2+
3+
_usage() {
4+
cat <<EOF
5+
6+
usage: $0 -s scenario -f config_file [-d directory]
7+
8+
Required
9+
-s scenario : scenario to run
10+
-f config_file : a file with list of scenarios
11+
12+
Optional
13+
-d directory : directory in which config_file may be found
14+
15+
config_file format
16+
==================
17+
File should be space-separated list of columns:
18+
19+
scenario: any one word that forms the key
20+
class: a valid CDM class to invoke, e.g. datastax.astra.migrate.Migrate
21+
properties: path to a valid CDM .properties file, there can be different files for each scenario
22+
23+
Example file:
24+
-------------------------
25+
migrateData datastax.astra.migrate.Migrate /smoke/01_basic_kvp/migrate.properties
26+
validateData datastax.astra.migrate.DiffData /smoke/01_basic_kvp/migrate.properties
27+
-------------------------
28+
29+
Output will be put into the same directory as the .properties file named like:
30+
cdm.scenario.out (stdout)
31+
cdm.scenario.err (stderr)
32+
33+
So the above example would generate files:
34+
cdm.migrateData.out
35+
cdm.migrateData.err
36+
cdm.validateData.out
37+
cdm.validateData.err
38+
39+
EOF
40+
exit 1
41+
}
42+
43+
while getopts "s:f:d:" opt; do
44+
case $opt in
45+
s) SCENARIO="$OPTARG"
46+
;;
47+
f) CONFIG_FILENAME="$OPTARG"
48+
;;
49+
d) CONFIG_DIR="$OPTARG"
50+
;;
51+
?) echo "ERROR invalid option was specified - ${OPTARG}"
52+
_usage
53+
;;
54+
esac
55+
done
56+
57+
argErrors=0
58+
if [[ -z "$CONFIG_FILENAME" ]]; then
59+
echo "missing -f config_file"
60+
argErrors=1
61+
else
62+
if [[ -z "${CONFIG_DIR}" ]]; then
63+
CONFIG_FILE="${CONFIG_FILENAME}"
64+
else
65+
CONFIG_FILE="${CONFIG_DIR}/${CONFIG_FILENAME}"
66+
fi
67+
if [[ ! -r ${CONFIG_FILE} ]]; then
68+
echo "config file ${CONFIG_FILE} not found, or is not readable"
69+
argErrors=1
70+
fi
71+
fi
72+
73+
if [[ -z "$SCENARIO" ]]; then
74+
echo "missing -s scenario"
75+
argErrors=1
76+
else
77+
if [[ $(egrep -c '^'$SCENARIO' ' $CONFIG_FILE) -ne 1 ]]; then
78+
echo "scenario not found in: ${CONFIG_FILE}"
79+
argErrors=1
80+
else
81+
CLASS=$(egrep '^'$SCENARIO' ' $CONFIG_FILE | awk '{print $2}')
82+
PROPERTIES=$(egrep '^'$SCENARIO' ' $CONFIG_FILE | awk '{print $3}')
83+
84+
if [[ ! -r "${PROPERTIES}" ]]; then
85+
echo ".properties file not found: ${PROPERTIES}"
86+
argErrors=1
87+
fi
88+
fi
89+
fi
90+
if [ $argErrors -ne 0 ]; then
91+
_usage
92+
fi
93+
94+
spark-submit --properties-file "${PROPERTIES}" --master "local[*]" --class ${CLASS} /local/cassandra-data-migrator.jar

‎SIT/common.sh

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
DOCKER_CASS=cdm-sit-cass
2+
DOCKER_CDM=cdm-sit-cdm
3+
CASS_USERNAME=cassandra
4+
CASS_PASSWORD=cassandra
5+
KEYSPACES="source target"
6+
CDM_DIRECTORY=/local
7+
CDM_JARFILE=cassandra-data-migrator.jar
8+
9+
_info() {
10+
echo "INFO $*"
11+
}
12+
13+
_warn() {
14+
echo "WARN $*"
15+
}
16+
17+
_error() {
18+
echo "ERROR $*"
19+
}
20+
21+
_fatal() {
22+
echo "FATAL $*"
23+
exit 2
24+
}

0 commit comments

Comments
 (0)