Update contributing test setup (#466)

add additional steps to test setup instructions

Author: ap76ap

CONTRIBUTING.md

@@ -100,7 +100,7 @@ You can choose from three different environments to test your fixes/changes, bas
 
 * To run a specific test function:
 
-``pytest tests/test_moto::test_get_bucket_region_succeed``
+``pytest tests/test_moto.py::test_get_bucket_region_succeed``
 
 * To run all mocked test functions (Using 8 parallel processes):
 
@@ -121,6 +121,10 @@ You can choose from three different environments to test your fixes/changes, bas
 
 ``pip install -r requirements-dev.txt``
 
+* [OPTIONAL] Set AWS_DEFAULT_REGION to define the region the Data Lake Test envrioment will deploy into. You may want to choose a region which you don't currently use:
+
+``export AWS_DEFAULT_REGION=ap-northeast-1``
+
 * Go to the ``cloudformation`` directory
 
 ``cd cloudformation``
@@ -145,10 +149,16 @@ You can choose from three different environments to test your fixes/changes, bas
 
 ``pytest -n 8 tests/test_athena*``
 
+* [OPTIONAL] To remove the base test environment cloud formation stack post testing:
+
+``./cloudformation/delete-base.sh``
+
 ### Full test environment
 
 **DISCLAIMER**: Make sure you know what you are doing. These steps will charge some services on your AWS account and require a minimum security skill to keep your environment safe.
 
+**DISCLAIMER**: This environment contains Aurora MySQL, Aurora PostgreSQL and Redshift (single-node) clusters which will incur cost while running.
+
 * Pick up a Linux or MacOS.
 * Install Python 3.6, 3.7 and 3.8
 * Fork the AWS Data Wrangler repository and clone that into your development environment
@@ -160,16 +170,27 @@ You can choose from three different environments to test your fixes/changes, bas
 
 ``pip install -r requirements-dev.txt``
 
+* [OPTIONAL] Set AWS_DEFAULT_REGION to define the region the Full Test envrioment will deploy into. You may want to choose a region which you don't currently use:
+
+``export AWS_DEFAULT_REGION=ap-northeast-1``
+
 * Go to the ``cloudformation`` directory
 
 ``cd cloudformation``
 
-* Deploy the Cloudformation templates `base.yaml` and `databases.yaml`
+* Deploy the Cloudformation templates `base.yaml` and `databases.yaml`. This step could take about 15 minutes to deploy.
 
 ``./deploy-base.sh``
 ``./deploy-databases.sh``
 
 * Go to the `EC2 -> SecurityGroups` console, open the `aws-data-wrangler-*` security group and configure to accept your IP from any TCP port.
+  - Alternatively run:
+  
+  ``./security-group-databases-add-local-ip.sh``
+  
+  - Check local IP was applied:
+  
+  ``./security-group-databases-check.sh``
 
 ``P.S Make sure that your security group will not be open to the World! Configure your security group to only give access for your IP.``
 
@@ -185,12 +206,22 @@ You can choose from three different environments to test your fixes/changes, bas
 
 * To run a specific test function:
 
-``pytest tests/test_athena_parquet.py::test_parquet_catalog``
+``pytest tests/test_db.py::test_sql``
+
+* To run all database test functions (Using 8 parallel processes):
+
+``pytest -n 8 tests/test_db.py``
 
 * To run all data lake test functions for all python versions (Only if Amazon QuickSight is activated):
 
 ``./test.sh``
 
+* [OPTIONAL] To remove the base test environment cloud formation stack post testing:
+
+``./cloudformation/delete-base.sh``
+
+``./cloudformation/delete-databases.sh``
+
 ## Recommended Visual Studio Code Recommended settings
 
 ```json

cloudformation/databases.yaml

@@ -408,3 +408,8 @@ Outputs:
         - AuroraInstanceMysql
         - Endpoint.Address
     Description: Mysql Address
+  DatabaseSecurityGroupId:
+    Value:
+      Fn::GetAtt:
+        - DatabaseSecurityGroup
+        - GroupId

cloudformation/delete-base.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -e
+
+# Deploying
+aws cloudformation delete-stack \
+  --stack-name aws-data-wrangler-base

cloudformation/delete-databases.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -e
+
+# Deleting
+aws cloudformation delete-stack \
+  --stack-name aws-data-wrangler-databases

cloudformation/security-group-databases-add-local-ip.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -e
+
+# Get my current IP address
+LOCALIP=`host myip.opendns.com resolver1.opendns.com | grep myip | awk '{print $4}'`
+
+# Get security group ID
+SGID=`aws cloudformation  describe-stacks --stack-name aws-data-wrangler-databases --query "Stacks[0].Outputs[?OutputKey=='DatabaseSecurityGroupId'].OutputValue" --output text`
+
+# Update Security Group with local ip
+aws ec2 authorize-security-group-ingress \
+  --group-id ${SGID} \
+  --protocol all \
+  --port -1 \
+  --cidr ${LOCALIP}/32

cloudformation/security-group-databases-check.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -e
+
+# Get security group ID
+SGID=`aws cloudformation  describe-stacks --stack-name aws-data-wrangler-databases --query "Stacks[0].Outputs[?OutputKey=='DatabaseSecurityGroupId'].OutputValue" --output text`
+
+# Check to see current setting
+aws ec2 describe-security-groups --group-id ${SGID}