* Add CDK test infrastructure (#706)

* Delete CFN template-based test infrastructure

* Update docs

* Replace db password string with a secret value

Author: kukushking

.github/workflows/static-checking.yml

@@ -28,10 +28,6 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install -U -r requirements-dev.txt
-      - name: CloudFormation Lint - Base
-        run: cfn-lint -t cloudformation/base.yaml
-      - name: CloudFormation Lint - Databases
-        run: cfn-lint -t cloudformation/databases.yaml
       - name: mypy check
         run: mypy awswrangler
       - name: Flake8 Lint

.gitignore

@@ -1,5 +1,5 @@
 # Byte-compiled / optimized / DLL files
-__pycache__/
+*__pycache__/
 *.py[cod]
 *$py.class
 
@@ -47,7 +47,7 @@ nosetests.xml
 coverage.xml
 *.cover
 .hypothesis/
-.pytest_cache/
+*.pytest_cache/
 test-reports/
 
 # Translations
@@ -90,9 +90,8 @@ celerybeat-schedule
 
 # Environments
 .env
-.venv
-.venv2
-env/
+*venv*/
+*env/
 venv*/
 ENV/
 env.bak/
@@ -145,3 +144,11 @@ building/lambda/arrow
 /docs/build/
 /docs/source/_build/
 /docs/source/stubs/
+
+# Swap
+*.swp
+
+# CDK
+*package-lock.json
+*.cdk.staging
+*cdk.out

CONTRIBUTING.md

@@ -137,21 +137,29 @@ or
 
 ``pip install -r requirements-dev.txt``
 
-* [OPTIONAL] Set AWS_DEFAULT_REGION to define the region the Data Lake Test envrioment will deploy into. You may want to choose a region which you don't currently use:
+* Go to the ``test_infra`` directory
+
+``cd test_infra``
+
+* Install CDK dependencies:
+
+``pip install -r requirements.txt``
+
+* [OPTIONAL] Set AWS_DEFAULT_REGION to define the region the Data Lake Test environment will deploy into. You may want to choose a region which you don't currently use:
 
 ``export AWS_DEFAULT_REGION=ap-northeast-1``
 
-* Go to the ``cloudformation`` directory
+* Go to the ``scripts`` directory
 
-``cd cloudformation``
+``cd scripts``
 
 * Deploy the Cloudformation template `base.yaml`
 
 ``./deploy-base.sh``
 
 * Return to the project root directory
 
-``cd ..``
+``cd ../../``
 
 * Run the validation script:
 
@@ -167,7 +175,7 @@ or
 
 * [OPTIONAL] To remove the base test environment cloud formation stack post testing:
 
-``./cloudformation/delete-base.sh``
+``./test_infra/scripts/delete-base.sh``
 
 ### Full test environment
 
@@ -186,13 +194,21 @@ or
 
 ``pip install -r requirements-dev.txt``
 
+* Go to the ``test_infra`` directory
+
+``cd test_infra``
+
+* Install CDK dependencies:
+
+``pip install -r requirements.txt``
+
 * [OPTIONAL] Set AWS_DEFAULT_REGION to define the region the Full Test envrioment will deploy into. You may want to choose a region which you don't currently use:
 
 ``export AWS_DEFAULT_REGION=ap-northeast-1``
 
-* Go to the ``cloudformation`` directory
+* Go to the ``scripts`` directory
 
-``cd cloudformation``
+``cd scripts``
 
 * Deploy the Cloudformation templates `base.yaml` and `databases.yaml`. This step could take about 15 minutes to deploy.
 
@@ -212,7 +228,7 @@ or
 
 * Return to the project root directory
 
-``cd ..``
+``cd ../../``
 
 * [OPTIONAL] If you intend to run all test, you also need to make sure that you have Amazon QuickSight activated and your AWS user must be register on that.
 
@@ -234,9 +250,9 @@ or
 
 * [OPTIONAL] To remove the base test environment cloud formation stack post testing:
 
-``./cloudformation/delete-base.sh``
+``./test_infra/scripts/delete-base.sh``
 
-``./cloudformation/delete-databases.sh``
+``./test_infra/scripts/delete-databases.sh``
 
 ## Recommended Visual Studio Code Recommended setting