Skip to content

Commit 17eec16

Browse files
feat: add integration tests (#69)
* feat: add integration tests * add eoapi-cdk upgrade command and pip cache * test commit, to revert * Revert "test commit, to revert" This reverts commit aab0acb. * update workflow * update workflow to run when release happens, move tests here * instead of cloning the eoapi-template repo, copy the code here so that we dont depend on an external repository. Modify accordingly the tests folder, with two subfolders (tests and cdk code) and the top folder renamed to integration_tests * trailing wsp * run action with push to try * move the trigger of integration tests to distribute, right before release * feat!: custom runtime for bootstrapper and custom runtimes from Dockerfile for all apps BREAKING CHANGE: clients need to provide aws_lambda.AssetCode to configure their apps. Solely the python application and the requirements.txt file is not supported anymore. * fix a couple bugs found in the first changes * avoid maintaining custom interfaces for configurable lambda properties. Allow the user to provide anything and let the CDK method raise error and overwrite values defined within our construct. Make this clear in the documentation * expose bootstrapper props in pgstacdatabase construct constructor * merge database and boostrapper files to solve casting bug * bump and harmonize pypgstac to 0.7.10 across apps * bump cachetools * some changes to allow for less security * bump python to 3.11 * change base image for bootstrapper to use python 311 * fix linting issues * move integration tests to step before release, improve naming of workflows * lint * fix moto requirement * test to fix deployment : try adding s3 endpoint and force allow public subnet * lint and make lambda functions more configurable * moving deploy to a separate workflow * remove useless dependencies in deployment tests, turn on pull request trigger to check the action works * when tearing down the infrastructure, synthesize the cloud formation assets into another directory to avoid conflicts * update readmes and revive the artifact download in python distribution --------- Co-authored-by: emileten <[email protected]>
1 parent 1e39ddb commit 17eec16

File tree

27 files changed

+468
-45
lines changed

27 files changed

+468
-45
lines changed

.github/workflows/build.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ on:
1212
required: false
1313
DS_RELEASE_BOT_PRIVATE_KEY:
1414
required: false
15-
15+
1616
jobs:
1717
build_and_package:
1818
name: Build and package
@@ -22,7 +22,7 @@ jobs:
2222

2323
- uses: actions/setup-node@v3
2424
with:
25-
node-version: 16
25+
node-version: 18
2626
cache: "npm"
2727

2828
- name: Install Dependencies
@@ -66,7 +66,7 @@ jobs:
6666
private_key: ${{ secrets.DS_RELEASE_BOT_PRIVATE_KEY }}
6767

6868
- name: Maybe Release 🚀
69-
if: ${{ inputs.release }}
69+
if: "${{ inputs.release }}"
7070
run: |
7171
npm run semantic-release
7272
env:

.github/workflows/test.yaml renamed to .github/workflows/build_and_release.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Test & Build
1+
name: Build & try to release
22

33
on:
44
push:

.github/workflows/deploy.yaml

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: Test deployment
2+
3+
on:
4+
merge_group:
5+
branches: [ main ]
6+
7+
jobs:
8+
build_package_and_deploy:
9+
name: Build, package and deploy
10+
runs-on: ubuntu-latest
11+
timeout-minutes: 60
12+
env:
13+
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION_DEPLOY }}
14+
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID_DEPLOY }}
15+
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY_DEPLOY }}
16+
AWS_DEFAULT_ACCOUNT: ${{ secrets.AWS_ACCOUNT_ID }}
17+
steps:
18+
- uses: actions/checkout@v3
19+
20+
- uses: actions/setup-node@v3
21+
with:
22+
node-version: 18
23+
cache: "npm"
24+
25+
- name: Install Dependencies
26+
run: npm ci
27+
28+
- name: Compile project
29+
run: npm run build
30+
31+
- name: Generate distribution packages
32+
run: npm run package
33+
34+
35+
- name: Install deployment environment
36+
id: install_deploy_env
37+
run: |
38+
# install deployment environment with eoapi-cdk from build
39+
python -m venv .deployment_venv
40+
source .deployment_venv/bin/activate
41+
pip install dist/python/*.gz
42+
cd integration_tests/cdk
43+
pip install -r requirements.txt
44+
npm install
45+
deactivate
46+
cd -
47+
48+
49+
- name: Deploy test stack
50+
id: deploy_step
51+
run: |
52+
source .deployment_venv/bin/activate
53+
54+
# synthesize the stack
55+
cd integration_tests/cdk
56+
npx cdk synth --debug --all --require-approval never
57+
58+
# deploy the stack
59+
npx cdk deploy --ci --all --require-approval never
60+
deactivate
61+
cd -
62+
63+
- name: Tear down any infrastructure
64+
if: always()
65+
run: |
66+
cd integration_tests/cdk
67+
# run this only if we find a 'cdk.out' directory, which means there might be things to tear down
68+
if [ -d "cdk.out" ]; then
69+
cd -
70+
source .deployment_venv/bin/activate
71+
cd integration_tests/cdk
72+
# see https://github.com/aws/aws-cdk/issues/24946
73+
# this didn't work : rm -f cdk.out/synth.lock
74+
# so we just duplicate the cdk output to cdk-destroy.out
75+
npx cdk destroy --output cdk-destroy.out --ci --all --force
76+
fi

.github/workflows/distribute.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,12 @@ jobs:
1313
runs-on: ubuntu-latest
1414
needs: package
1515
steps:
16+
1617
- uses: actions/download-artifact@v3
1718
with:
1819
name: python
1920
path: dist
20-
21+
2122
- run: pip install twine
2223

2324
- run: twine upload dist/*

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,6 @@ docs
88
__pycache__
99
.venv
1010
.tox
11+
tests/*.egg*
12+
tests/*venv*
13+
tests/__pycache__

README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,6 @@ Versioning is automatically handled via [Conventional Commits](https://www.conve
5454
_Warning_: If you rebase `main`, you must ensure that the commits referenced by tags point to commits that are within the `main` branch. If a commit references a commit that is no longer on the `main` branch, Semantic Release will fail to detect the correct version of the project. [More information](https://github.com/semantic-release/semantic-release/issues/1121#issuecomment-517945233).
5555

5656

57+
## Tests
58+
59+
Each pull request to `main` is added to a [merge queue](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue#triggering-merge-group-checks-with-github-actions) so that a "deployment test" workflow can run before the merge actually happens. If the deployment fails, the merge is cancelled. Here is [the definition of this workflow](https://github.com/developmentseed/eoapi-cdk/blob/main/.github/workflows/deploy.yaml) and the [tests definition](https://github.com/developmentseed/eoapi-cdk/blob/main/tests).

integration_tests/cdk/README.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
2+
# Deployment CDK code for eoapi-cdk deployment tests
3+
4+
This is a wrapper CDK code that is used to test a deployment of the `eoapi-cdk` constructs before a release happens.
5+
6+
## Requirements
7+
8+
- python
9+
- docker
10+
- node
11+
- AWS credentials environment variables configured to point to an account.
12+
13+
## Installation
14+
15+
Install python dependencies with
16+
17+
```
18+
python -m venv .venv
19+
source .venv/bin/activate
20+
python -m pip install -r requirements.txt
21+
```
22+
23+
Install the latest `eoapi-cdk` either from PyPI:
24+
25+
```
26+
pip install eoapi-cdk
27+
```
28+
29+
Or alternatively, compile and package from the root of this repository to get the python version of the constructs locally.
30+
31+
Also install node dependencies with
32+
33+
```
34+
npm install
35+
```
36+
37+
Verify that the `cdk` CLI is available. Since `aws-cdk` is installed as a local dependency, you can use the `npx` node package runner tool, that comes with `npm`.
38+
39+
```
40+
npx cdk --version
41+
```
42+
43+
## Deployment
44+
45+
First, synthesize the app
46+
47+
```
48+
npx cdk synth --all
49+
```
50+
51+
Then, deploy
52+
53+
```
54+
npx cdk deploy --all --require-approval never
55+
```

integration_tests/cdk/app.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
from aws_cdk import App
2+
3+
from config import build_app_config
4+
from eoapi_template import pgStacInfra, vpc
5+
6+
app = App()
7+
8+
app_config = build_app_config()
9+
10+
vpc_stack = vpc.VpcStack(scope=app, app_config=app_config)
11+
12+
pgstac_infra_stack = pgStacInfra.pgStacInfraStack(
13+
scope=app,
14+
vpc=vpc_stack.vpc,
15+
app_config=app_config,
16+
)
17+
app.synth()

integration_tests/cdk/cdk.json

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"app": "python3 app.py",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"requirements*.txt",
11+
"source.bat",
12+
"**/*.pyc",
13+
"**/*.tmp",
14+
"**/__pycache__",
15+
"tests",
16+
"scripts",
17+
"*venv"
18+
]
19+
},
20+
"context": {
21+
"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
22+
"@aws-cdk/core:stackRelativeExports": true,
23+
"@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
24+
"@aws-cdk/aws-lambda:recognizeVersionProps": true,
25+
"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/core:target-partitions": [
28+
"aws",
29+
"aws-cn"
30+
]
31+
}
32+
}

integration_tests/cdk/config.py

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
from typing import Dict
2+
3+
import pydantic
4+
import yaml
5+
from pydantic_core.core_schema import FieldValidationInfo
6+
from pydantic_settings import BaseSettings, SettingsConfigDict
7+
8+
9+
class AppConfig(BaseSettings):
10+
model_config = SettingsConfigDict(
11+
env_file=".env"
12+
)
13+
aws_default_account: str = pydantic.Field(
14+
description="AWS account ID"
15+
)
16+
project_id: str = pydantic.Field(
17+
description="Project ID", default="eoapi-cdk"
18+
)
19+
stage: str = pydantic.Field(description="Stage of deployment", default="test")
20+
# because of its validator, `tags` should always come after `project_id` and `stage`
21+
tags: Dict[str, str] | None = pydantic.Field(
22+
description="""Tags to apply to resources. If none provided,
23+
will default to the defaults defined in `default_tags`.
24+
Note that if tags are passed to the CDK CLI via `--tags`,
25+
they will override any tags defined here.""",
26+
default=None,
27+
)
28+
db_instance_type: str = pydantic.Field(
29+
description="Database instance type", default="t3.micro"
30+
)
31+
db_allocated_storage: int = pydantic.Field(
32+
description="Allocated storage for the database", default=5
33+
)
34+
35+
@pydantic.field_validator("tags")
36+
def default_tags(cls, v, info: FieldValidationInfo):
37+
return v or {"project_id": info.data["project_id"], "stage": info.data["stage"]}
38+
39+
def build_service_name(self, service_id: str) -> str:
40+
return f"{self.project_id}-{self.stage}-{service_id}"
41+
42+
43+
def build_app_config() -> AppConfig:
44+
"""Builds the AppConfig object from config.yaml file if exists,
45+
otherwise use defaults"""
46+
try:
47+
with open("config.yaml") as f:
48+
print("Loading config from config.yaml")
49+
app_config = yaml.safe_load(f)
50+
app_config = (
51+
{} if app_config is None else app_config
52+
) # if config is empty, set it to an empty dict
53+
app_config = AppConfig(**app_config)
54+
except FileNotFoundError:
55+
# if no config at the expected path, using defaults
56+
app_config = AppConfig()
57+
58+
return app_config

0 commit comments

Comments
 (0)