-
Notifications
You must be signed in to change notification settings - Fork 110
Initial commit #367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Initial commit #367
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
26e6ee0
Initial commit
3d19ae1
Examples migrated to pytest testing - Step Functions
d1ea8b4
Examples migrated to pytest testing - README
1b0163c
Examples migrated to pytest testing - README 2
40585f5
Updating READMEs as per PR comments
da057e2
Splitting aws cli and pytest cases and adding Debug commands
dfeeb04
Deleting network host references and path fixes as per PR comments
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
259 changes: 259 additions & 0 deletions
259
python-test-samples/step-functions-local-helloworld/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,259 @@ | ||
[](https://img.shields.io/badge/Python-3.10-green) | ||
[](https://img.shields.io/badge/AWS-Step%20Functions-orange) | ||
[](https://img.shields.io/badge/Language-ASL-blue) | ||
[](https://img.shields.io/badge/Test-Pytest-red) | ||
[](https://img.shields.io/badge/Test-Local-red) | ||
|
||
# AWS Step Functions: Hello World with PyTest | ||
|
||
## Introduction | ||
|
||
This project demonstrates how to test AWS Step Functions state machines locally using Docker and PyTest. It focuses on a simple "Hello World" state machine with a Pass state, showing how to create and validate automated tests for AWS Step Functions locally. | ||
|
||
|
||
--- | ||
|
||
## Contents | ||
- [AWS Step Functions: Hello World with PyTest](#aws-step-functions-hello-world-with-pytest) | ||
- [Introduction](#introduction) | ||
- [Contents](#contents) | ||
- [Architecture Overview](#architecture-overview) | ||
- [Project Structure](#project-structure) | ||
- [Prerequisites](#prerequisites) | ||
- [Test Scenarios](#test-scenarios) | ||
- [About the Test Process](#about-the-test-process) | ||
- [Testing Workflows](#testing-workflows) | ||
- [Additional Resources](#additional-resources) | ||
|
||
--- | ||
|
||
## Architecture Overview | ||
<p align="center"> | ||
<img src="img/stepfunctions-helloworld.png" alt="Hello World Step Functions" width="300"/> | ||
</p> | ||
|
||
Components: | ||
- Simple Step Functions state machine with a Pass state | ||
- Testcontainers for container management | ||
- PyTest for automated testing | ||
- AWS Step Functions Local for local execution | ||
|
||
<p align="center"> | ||
<img src="img/stepfunctions-helloworld-states.png" alt="Hello World Step Functions - States" width="300"/> | ||
</p> | ||
|
||
--- | ||
|
||
## Project Structure | ||
brnkrygs marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
├── img/ | ||
│ ├── stepfunctions-helloworld-states.png _# Step Functions state flow_ | ||
│ └── stepfunctions-mock.png _# visual architecture diagram_ | ||
├── statemachine/ | ||
│ ├── local_testing.asl.json _# json file containing "Hello World" state machine definition_ | ||
│ └── test/valid_input.json _# json file containing "Hello World" state machine input_ | ||
├── tests/ | ||
│ ├── unit/src/test_step_functions_local.py _# python PyTest test definition_ | ||
│ ├── aws-stepfunctions-local-credentials.txt _# Step Functions Docker image environment vars_ | ||
│ └── requirements.txt _# pip requirments dependencies file_ | ||
└── README.md _# instructions file_ | ||
``` | ||
|
||
--- | ||
|
||
## Prerequisites | ||
- Docker | ||
- AWS cli (debugging) | ||
- Python 3.10 or newer | ||
- Basic understanding of Step Functions | ||
- Basic understanding of Amazon States Language (ASL) | ||
|
||
--- | ||
|
||
## Test Scenarios (end to end python test) | ||
|
||
### 1. Hello World | ||
- Tests the happy path where a single state in step funtions succeed on the first attempt | ||
- Used to validate the basic functionality of the state machine | ||
|
||
--- | ||
|
||
## About the Test Process | ||
|
||
The test process leverages PyTest fixtures to manage the lifecycle of the Step Functions Local container and the state machine: | ||
|
||
1. **Container Setup**: The `fixture_container` fixture downloads and starts the `amazon/aws-stepfunctions-local` container. | ||
|
||
2. **State Machine Creation**: The `fixture_sfn_client` fixture creates a Boto3 client for Step Functions and creates the state machine using the definition from `local_testing.asl.json`. | ||
|
||
3. **Test Execution**: When the test runs, it calls `execute_stepfunction` to start an execution of the state machine and monitor it until completion. | ||
|
||
4. **Validation**: The test verifies that: | ||
- The execution completed successfully | ||
- The HelloWorld state was properly exited | ||
|
||
5. **Cleanup**: After the test completes, the container is automatically shut down by the finalizer added in the `fixture_container`. | ||
|
||
--- | ||
|
||
## Testing Workflows | ||
|
||
### Setup Docker Environment | ||
|
||
> Make sure Docker engine is running before running the tests. | ||
|
||
```shell | ||
step-functions-local-lambda$ docker version | ||
Client: Docker Engine - Community | ||
Version: 24.0.6 | ||
API version: 1.43 | ||
(...) | ||
``` | ||
|
||
### Run the Unit Test - End to end python test | ||
|
||
> Set up the python environment | ||
|
||
``` shell | ||
step-functions-local-helloworld$ cd tests | ||
export AWS_ACCESS_KEY_ID='DUMMYIDEXAMPLE' | ||
export AWS_SECRET_ACCESS_KEY='DUMMYEXAMPLEKEY' | ||
export AWS_REGION='us-east-1' | ||
python3 -m venv venv | ||
source venv/bin/activate | ||
pip install --upgrade pip | ||
pip install -r requirements.txt | ||
``` | ||
|
||
#### Run the unit tests | ||
|
||
``` shell | ||
step-functions-local-helloworld/tests$ | ||
python3 -m pytest -s unit/src/test_step_functions_local.py | ||
``` | ||
|
||
Expected output: | ||
``` shell | ||
step-functions-local-helloworld/tests$ python3 -m pytest -s unit/src/test_step_functions_local.py | ||
========================================================== test session starts ========================================================== | ||
platform linux -- Python 3.10.12, pytest-8.4.1, pluggy-1.6.0 | ||
rootdir: /home/ubuntu/environment/serverless-test-samples-approach-2-pytest-distributed_Try3/step-functions-local-helloworld--OK/tests | ||
collected 1 item | ||
|
||
unit/src/test_step_functions_local.py Pulling image testcontainers/ryuk:0.8.1 | ||
Container started: ecdd9649b97d | ||
Waiting for container <Container: ecdd9649b97d> with image testcontainers/ryuk:0.8.1 to be ready ... | ||
Pulling image amazon/aws-stepfunctions-local | ||
Container started: d1bd33969e6b | ||
Waiting for container <Container: d1bd33969e6b> with image amazon/aws-stepfunctions-local to be ready ... | ||
. | ||
|
||
========================================================== 1 passed in 10.57s ========================================================== | ||
( | ||
``` | ||
|
||
#### Clean up section | ||
|
||
> clean pyenv environment | ||
|
||
```sh | ||
step-functions-local-helloworld/tests$ | ||
deactivate | ||
rm -rf venv/ | ||
``` | ||
|
||
> unsetting variables | ||
|
||
```sh | ||
unset AWS_ACCESS_KEY_ID | ||
unset AWS_SECRET_ACCESS_KEY | ||
unset AWS_REGION | ||
``` | ||
|
||
> cleanning docker | ||
|
||
```sh | ||
docker ps --filter "ancestor=amazon/aws-stepfunctions-local" -q | xargs -r docker kill | ||
docker rmi amazon/aws-stepfunctions-local | ||
``` | ||
|
||
#### Debug - PyTest Debugging | ||
|
||
For more detailed debugging in pytest: | ||
|
||
```sh | ||
# Run with verbose output | ||
python -m pytest -s -v unit/src/test_step_functions_local.py | ||
|
||
# Run with debug logging | ||
python -m pytest -s -v unit/src/test_step_functions_local.py --log-cli-level=DEBUG | ||
``` | ||
|
||
--- | ||
|
||
### Fast local development for Step Functions | ||
|
||
#### AWS CLI Commands for Manual Verification | ||
|
||
If you need to manually verify the state machine or execution details, you can use these commands: | ||
|
||
#### Configure environment variables: | ||
|
||
```sh | ||
step-functions-local-helloworld/tests$ | ||
brnkrygs marked this conversation as resolved.
Show resolved
Hide resolved
|
||
export AWS_ACCESS_KEY_ID='DUMMYIDEXAMPLE' | ||
export AWS_SECRET_ACCESS_KEY='DUMMYEXAMPLEKEY' | ||
export AWS_REGION='us-east-1' | ||
``` | ||
|
||
#### Set up state machine machine manually | ||
|
||
```sh | ||
# Runing step functions docker image in the background | ||
step-functions-local-helloworld/tests$ | ||
docker run -d \ | ||
--name stepfunctions -p 8083:8083 \ | ||
--env-file aws-stepfunctions-local-credentials.txt amazon/aws-stepfunctions-local | ||
``` | ||
|
||
```sh | ||
# Creating Hello World state machine | ||
aws stepfunctions create-state-machine --endpoint-url http://localhost:8083 \ | ||
--name "CRUDDynamoDB" --role-arn "arn:aws:iam::012345678901:role/DummyRole" \ | ||
--region us-east-1 \ | ||
--definition file://../statemachine/local_testing.asl.json | ||
``` | ||
|
||
#### Debug state machine - aws cli debugging | ||
|
||
```sh | ||
# Get state machine ARN | ||
aws stepfunctions list-state-machines --endpoint-url http://localhost:8083 | ||
|
||
# Check state machine definition | ||
aws stepfunctions describe-state-machine --endpoint-url http://localhost:8083 \ | ||
--state-machine-arn [STATE-MACHINE-ARN] | ||
enmata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Start state machine manually | ||
aws stepfunctions start-execution --endpoint-url http://localhost:8083 \ | ||
--state-machine-arn [STATE-MACHINE-ARN] | ||
|
||
# Check execution details | ||
aws stepfunctions describe-execution --endpoint http://localhost:8083 \ | ||
--execution-arn [STATE-MACHINE-EXECUTION-ARN] | ||
enmata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Check execution history | ||
aws stepfunctions get-execution-history --endpoint http://localhost:8083 \ | ||
--execution-arn [STATE-MACHINE-EXECUTION-ARN] | ||
enmata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
|
||
--- | ||
|
||
## Additional Resources | ||
enmata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- [Step Functions Local Guide](https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local.html) | ||
- [Amazon States Language Documentation](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html) | ||
- [AWS Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) | ||
- [PyTest Documentation](https://docs.pytest.org/) | ||
- [Testcontainers Python Documentation](https://testcontainers-python.readthedocs.io/en/latest/) | ||
|
||
[Top](#contents) |
Binary file added
BIN
+9.39 KB
...samples/step-functions-local-helloworld/img/stepfunctions-helloworld-states.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+51.8 KB
...n-test-samples/step-functions-local-helloworld/img/stepfunctions-helloworld.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 10 additions & 0 deletions
10
python-test-samples/step-functions-local-helloworld/statemachine/local_testing.asl.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
{ | ||
"Comment": "Hello World State Machine of the Amazon States Language using a Pass state", | ||
"StartAt": "HelloWorld", | ||
"States": { | ||
"HelloWorld": { | ||
"Type": "Pass", | ||
"End": true | ||
} | ||
} | ||
} |
6 changes: 6 additions & 0 deletions
6
python-test-samples/step-functions-local-helloworld/statemachine/test/valid_input.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"data": { | ||
}, | ||
"comments": "" | ||
} | ||
} |
3 changes: 3 additions & 0 deletions
3
...est-samples/step-functions-local-helloworld/tests/aws-stepfunctions-local-credentials.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
AWS_ACCESS_KEY_ID=DUMMYIDEXAMPLE | ||
AWS_SECRET_ACCESS_KEY=DUMMYEXAMPLEKEY | ||
AWS_DEFAULT_REGION=us-east-1 |
4 changes: 4 additions & 0 deletions
4
python-test-samples/step-functions-local-helloworld/tests/requirements.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
boto3>=1.26.79 | ||
pytest>=7.2.1 | ||
pathlib | ||
testcontainers |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.