Skip to content

Conversation

@yiyuan-he
Copy link
Contributor

@yiyuan-he yiyuan-he commented Jan 9, 2025

Issue description:
Runtime error in our sample app caused by implicit dependency. The initialization error was fixed in this PR but the runtime error from the app still persists which is why the main build is still failing.

Logs from the EKS playground cluster to reproduce issue.

❯ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
my-app-5c5c6b949f-9jq9q   1/1     Running   0          8s
my-app-5c5c6b949f-xwgd5   1/1     Running   0          22s

~ via  v17.0.2 via  v22.11.0 via  v3.8.18 on   (us-east-1)
❯ kubectl logs my-app-5c5c6b949f-9jq9q
Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init)
Error in sitecustomize; set PYTHONVERBOSE for traceback:
RuntimeError: Python 3.9 or later is required
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK
Error in sitecustomize; set PYTHONVERBOSE for traceback:
RuntimeError: Python 3.9 or later is required

125 static files copied to '/django_frontend_app/static'.
Error in sitecustomize; set PYTHONVERBOSE for traceback:
RuntimeError: Python 3.9 or later is required
Performing system checks...

System check identified no issues (0 silenced).
January 09, 2025 - 01:57:17
Django version 4.2.16, using settings 'django_frontend_service.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

Description of changes:
Pinned the urllib3 version to 2.2.3 in the requirements.txt.

Test plan:
Built the django frontend service app locally with the change and deployed to public ECR in my AWS account. Logged into EKS playground cluster and modified the sample app image to point to this ECR image and used the hash commit from this PR for the SDK version.

❯ kubectl logs my-app-5b4f546545-qql8w
Defaulted container "my-container" out of: my-container, opentelemetry-auto-instrumentation-python (init)
AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS.
AwsEksResourceDetector failed: HTTP Error 403: Forbidden
Configuration of configurator not loaded, aws_configurator already loaded
('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.')
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK
AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS.
AwsEksResourceDetector failed: HTTP Error 403: Forbidden
Configuration of configurator not loaded, aws_configurator already loaded
('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.')

125 static files copied to '/django_frontend_app/static'.
AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS.
AwsEksResourceDetector failed: HTTP Error 403: Forbidden
Configuration of configurator not loaded, aws_configurator already loaded
('Could not find the ORIG_HANDLER or _HANDLER in the environment variables. ', 'This instrumentation requires the OpenTelemetry Lambda extension installed.')
Performing system checks...

System check identified no issues (0 silenced).
January 09, 2025 - 04:53:13
Django version 4.2.17, using settings 'django_frontend_service.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

Rollback procedure:

Yes we can easily remove the line in the requirements.txt for the version pin and redeploy the ECR image.

Ensure you've run the following tests on your changes and include the link below:

To do so, create a test.yml file with name: Test and workflow description to test your changes, then remove the file for your PR. Link your test run in your PR description. This process is a short term solution while we work on creating a staging environment for testing.

NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL. See the needs keyword to run tests in succession.

  • Run Java EKS on e2e-playground in us-east-1 and eu-central-2
  • Run Python EKS on e2e-playground in us-east-1 and eu-central-2
  • Run metric limiter on EKS cluster e2e-playground in us-east-1 and eu-central-2
  • Run EC2 tests in all regions
  • Run K8s on a separate K8s cluster (check IAD test account for master node endpoints; these will change as we create and destroy clusters for OS patching)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mxiamxia mxiamxia merged commit 4d358c7 into aws-observability:main Jan 9, 2025
1 check passed
yiyuan-he added a commit that referenced this pull request Jan 9, 2025
mxiamxia pushed a commit that referenced this pull request Jan 9, 2025
This reverts commit 4d358c7.

*Issue description:*
Root cause was found to be another dependency dropping support for
python 3.8. This change is no longer needed.

More context
[here](aws-observability/aws-otel-python-instrumentation#311).

*Description of changes:*

*Rollback procedure:*

<Can we safely revert this commit if needed? If not, detail what must be
done to safely revert and why it is needed.>

*Ensure you've run the following tests on your changes and include the
link below:*

To do so, create a `test.yml` file with `name: Test` and workflow
description to test your changes, then remove the file for your PR. Link
your test run in your PR description. This process is a short term
solution while we work on creating a staging environment for testing.

NOTE: TESTS RUNNING ON A SINGLE EKS CLUSTER CANNOT BE RUN IN PARALLEL.
See the
[needs](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idneeds)
keyword to run tests in succession.
- Run Java EKS on `e2e-playground` in us-east-1 and eu-central-2
- Run Python EKS on `e2e-playground` in us-east-1 and eu-central-2
- Run metric limiter on EKS cluster `e2e-playground` in us-east-1 and
eu-central-2
- Run EC2 tests in all regions
- Run K8s on a separate K8s cluster (check IAD test account for master
node endpoints; these will change as we create and destroy clusters for
OS patching)

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants