pin urllib3 version in docker setup #310

yiyuan-he · 2025-01-08T22:45:35Z

Issue #, if available:
Attempt to pin urllib3 version to fix python 3.8 runtime issue.

Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init)
Error in sitecustomize; set PYTHONVERBOSE for traceback:
RuntimeError: Python 3.9 or later is required
Traceback (most recent call last):
  File "manage.py", line 23, in <module>
    main()
  File "manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
.
.
.
 File "/usr/local/lib/python3.8/site-packages/botocore/compat.py", line 32, in <module>
    from urllib3 import exceptions
  File "/otel-auto-instrumentation-python/urllib3/__init__.py", line 14, in <module>
    from . import exceptions
  File "/otel-auto-instrumentation-python/urllib3/exceptions.py", line 26, in <module>
    _TYPE_REDUCE_RESULT = tuple[typing.Callable[..., object], tuple[object, ...]]
TypeError: 'type' object is not subscriptable

Test plan:
Tested locally on playground EKS cluster and validated the change fixes the startup runtime issue. Will monitor EK2 workflow after PR is merged to ensure it passes there as well.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Dockerfile

*Issue description:* Runtime error in our sample app caused by implicit dependency. The initialization error was fixed in this [PR](aws-observability/aws-otel-python-instrumentation#310) but the runtime error from the app still persists which is why the [main build is still failing](https://github.com/aws-observability/aws-otel-python-instrumentation/actions/runs/12681094567/job/35344577633). Logs from the EKS playground cluster to reproduce issue. ``` ❯ kubectl get pods NAME READY STATUS RESTARTS AGE my-app-5c5c6b949f-9jq9q 1/1 Running 0 8s my-app-5c5c6b949f-xwgd5 1/1 Running 0 22s ~ via  v17.0.2 via  v22.11.0 via  v3.8.18 on  (us-east-1) ❯ kubectl logs my-app-5c5c6b949f-9jq9q Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init) Error in sitecustomize; set PYTHONVERBOSE for traceback: RuntimeError: Python 3.9 or later is required Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying sessions.0001_initial... OK Error in sitecustomize; set PYTHONVERBOSE for traceback: RuntimeError: Python 3.9 or later is required 125 static files copied to '/django_frontend_app/static'. Error in sitecustomize; set PYTHONVERBOSE for traceback: RuntimeError: Python 3.9 or later is required Performing system checks... System check identified no issues (0 silenced). January 09, 2025 - 01:57:17 Django version 4.2.16, using settings 'django_frontend_service.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. ``` *Description of changes:* Pinned the urllib3 version to `2.2.3` in the `requirements.txt`. *Test plan:* Built the django frontend service app locally with the change and deployed to public ECR in my AWS account. Logged into EKS playground cluster and modified the sample app image to point to this ECR image and used the hash commit from this [PR](aws-observability/aws-otel-python-instrumentation#310) for the SDK version. ``` ❯ kubectl logs my-app-5b4f546545-qql8w Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init) AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying sessions.0001_initial... OK AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') 125 static files copied to '/django_frontend_app/static'. AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') Performing system checks... System check identified no issues (0 silenced). January 09, 2025 - 04:53:13 Django version 4.2.17, using settings 'django_frontend_service.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. ``` *Rollback procedure:* Yes we can easily remove the line in the `requirements.txt` for the version pin and redeploy the ECR image. *Ensure you've run the following tests on your changes and include the link below:* To do so, create a `test.yml` file with `name: Test` and workflow description to test your changes, then remove the file for your PR. Link your test run in your PR description. This process is a short term solution while we work on creating a staging environment for testing. NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL. See the [needs](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idneeds) keyword to run tests in succession. - Run Java EKS on `e2e-playground` in us-east-1 and eu-central-2 - Run Python EKS on `e2e-playground` in us-east-1 and eu-central-2 - Run metric limiter on EKS cluster `e2e-playground` in us-east-1 and eu-central-2 - Run EC2 tests in all regions - Run K8s on a separate K8s cluster (check IAD test account for master node endpoints; these will change as we create and destroy clusters for OS patching) By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

*Issue #, if available:* We found another issue with a dependency (`setuptools`) dropping Python 3.8 support. [Dependency issue with urllib3 PR](#310). ``` File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/__init__.py", line 22, in <module> from pkg_resources import iter_entry_points File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 843, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/otel-auto-instrumentation-python/pkg_resources/__init__.py", line 25, in <module> raise RuntimeError("Python 3.9 or later is required") RuntimeError: Python 3.9 or later is required ``` https://github.com/pypa/setuptools/blame/main/pkg_resources/__init__.py#L24 *Description of changes:* Pinned most recent compatible version of `setuptools` in our `pyproject.toml`. *Test plan:* Pushed this fixed version of SDK to an ECR repo and tested in the playground cluster to confirm the runtime errors are resolved. ``` ❯ kubectl get pods NAME READY STATUS RESTARTS AGE my-app-5c5c6b949f-htjj2 1/1 Running 0 7s my-app-5c5c6b949f-zk9nn 1/1 Running 0 7s my-app-76fbfbd7fc-mb4ft 1/1 Terminating 0 7m12s my-app-76fbfbd7fc-tpql7 1/1 Terminating 0 7m12s appsignals_django_uwsgi_sample/myproject on  main [?] via  desktop-linux via  v3.8.18 on  (us-east-1) ❯ kubectl logs my-app-5c5c6b949f-htjj2 Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init) AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying sessions.0001_initial... OK AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') 125 static files copied to '/django_frontend_app/static'. AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS. AwsEksResourceDetector failed: HTTP Error 403: Forbidden Configuration of configurator not loaded, aws_configurator already loaded ('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.') Performing system checks... System check identified no issues (0 silenced). January 09, 2025 - 19:28:09 Django version 4.2.16, using settings 'django_frontend_service.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. ``` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

pin urllib3 version in docker setup

c043af2

yiyuan-he requested a review from a team as a code owner January 8, 2025 22:45

mxiamxia previously approved these changes Jan 8, 2025

View reviewed changes

mxiamxia reviewed Jan 8, 2025

View reviewed changes

Dockerfile Show resolved Hide resolved

srprash reviewed Jan 8, 2025

View reviewed changes

Dockerfile Show resolved Hide resolved

add comment to explain context of urllib version restriction

f8cd768

yiyuan-he dismissed mxiamxia’s stale review via f8cd768 January 8, 2025 23:28

apply lint

70caca0

mxiamxia approved these changes Jan 8, 2025

View reviewed changes

srprash approved these changes Jan 8, 2025

View reviewed changes

yiyuan-he merged commit 8345735 into aws-observability:main Jan 8, 2025
12 checks passed

yiyuan-he mentioned this pull request Jan 9, 2025

pin urllib3 version in sample app aws-observability/aws-application-signals-test-framework#349

Merged

yiyuan-he mentioned this pull request Jan 9, 2025

pin setuptools version to 75.2.0 #311

Merged

yiyuan-he deleted the fix-docker-build branch March 3, 2025 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

pin urllib3 version in docker setup #310

pin urllib3 version in docker setup #310

Uh oh!

yiyuan-he commented Jan 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pin urllib3 version in docker setup #310

pin urllib3 version in docker setup #310

Uh oh!

Conversation

yiyuan-he commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yiyuan-he commented Jan 8, 2025 •

edited

Loading