-
Notifications
You must be signed in to change notification settings - Fork 783
Description
Running ./mwaa-local-env package-requirements will always result in an exit code of 0, even on failure.
Steps to Reproduce:
Note: All steps were performed using v2.8.1, but the behavior should be the same for all versions that include this function (v2.2.2+).
-
Modify
requirements.txtto force a failure. E.g. addapache-airflow-providers-datadog==3.5.0to therequirements.txtincluded in the repo. -
Run
./mwaa-local-env package-requirementsand check exit code.Click here for shell output
➜ aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env package-requirements Container amazon/mwaa-local:2_8 exists. Skipping build Packaging requirements.txt into plugins Collecting apache-airflow-providers-snowflake==5.2.1 Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB) Collecting apache-airflow-providers-mysql==5.5.1 Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB) ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies. The conflict is caused by: The user requested apache-airflow-providers-datadog==3.5.0 The user requested (constraint) apache-airflow-providers-datadog==3.5.1 To fix this you could try to: 1. loosen the range of package versions you've specified 4. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts [notice] A new release of pip available: 22.3.1 -> 24.2 [notice] To update, run: pip install --upgrade pip updating: README.md (deflated 37%) ➜ aws-mwaa-local-runner git:(v2.8.1) ✗ echo $? 0
Note that ./mwaa-local-env test-requirements does have a proper non-zero exit code on failure. However both commands should have a non-zero exit code on failure.
Click here for shell output
➜ aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env test-requirements
Container amazon/mwaa-local:2_8 exists. Skipping build
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"
Installing requirements.txt
Collecting apache-airflow-providers-snowflake==5.2.1
Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB)
Collecting apache-airflow-providers-mysql==5.5.1
Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB)
ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested apache-airflow-providers-datadog==3.5.0
The user requested (constraint) apache-airflow-providers-datadog==3.5.1
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
[notice] A new release of pip available: 22.3.1 -> 24.2
[notice] To update, run: pip install --upgrade pip
➜ aws-mwaa-local-runner git:(v2.8.1) ✗ echo $?
1Cause:
The issue stems from the package_requirements() function in entrypoint.sh. Specifically, there is no check that pip3 download -r ... completed successfully. The script will just attempt to zip whatever was downloaded and continue on it's way.
Fix:
I'm working on a fix for this and will open a PR to v2.9.2 when it is ready. I kindly request the fix be merged to lower branches (either all v2.2.2+ or at least to the branches for versions that are still supported).
Thank you!