Fix Gevent patch regression with correct import order (aws-observability#522)

### Issue
[This
PR](aws-observability#218)
first fixed the Gevent monkey patching issue by calling the patch
operation when the ADOT Python starts up and before the modules are
loaded.
Later on in v0.10.0, [this
commit](aws-observability@aed584f#diff-8edce37d4d49a64bdb9e89ba42e612aeea53757a4f317a94b80b883ce54008e9)
broke the above fix by adding
the line `from
amazon.opentelemetry.distro.aws_opentelemetry_configurator` which
transitively imports the `requests` library before the gevent patch
operation is called, causing inconsistencies in the patched libraries
resulting in seeing the `RecursionError: maximum recursion depth
exceeded` error reappearing.

### Changes
1. Adding back the earlier logic to run the gevent's monkey patch (only
if `gevent` is installed) which was removed in [this
commit](aws-observability@1326576#diff-acd1d4a6916a3593419e6117394fe1c7d07b69d1562917dc0a2af45cd7ee8526)
to unblock development.
2. Refactoring the patch into its own file and adding comprehensive
comments to avoid the same issue in future.
3. Updated unit tests to mock the `gevent` module and validate only the
ADOT logic without running the gevent monkey patch. The actual behavior
of the patch should be tested in a higher level test such as a contract
test.

### Testing
Did a manual E2E testing using a sample flask application. Confirmed the
following:
- Saw `RecursionError` in the application with ADOT Python v0.10.0 
- No such error when using ADOT with this fix.

#### Setup
##### Sample app
```python
from flask import Flask, request, jsonify
import logging
import sys
from datetime import datetime

import requests

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

app = Flask(__name__)
logger = logging.getLogger(__name__)

@app.route('/requests')
def make_request():
    res = requests.get('https://httpbin.org/get', timeout=5)
    return jsonify({
        'status': 'success',
        'status_code': res.status_code,
        'url': res.url,
        'timestamp': datetime.now().isoformat()
    })

if __name__ == '__main__':
    app.run(debug=False, host='0.0.0.0', port=5001)

```

##### Running with problematic version (see the `MonkeyPatchWarning`)
```shell
((venv) ) ➜  flask git:(fix_gevent_regression) ✗ opentelemetry-instrument gunicorn -w 1 -k gevent --bind 0.0.0.0:5001 app:app
/Users/srprash/aws-otel-python-instrumentation/venv/lib/python3.12/site-packages/amazon/opentelemetry/distro/patches/_instrumentation_patch.py:36: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See gevent/gevent#1016. Modules that had direct imports (NOT patched): ['urllib3.util.ssl_ (/Users/srprash/aws-otel-python-instrumentation/venv/lib/python3.12/site-packages/urllib3/util/ssl_.py)', 'urllib3.util (/Users/srprash/aws-otel-python-instrumentation/venv/lib/python3.12/site-packages/urllib3/util/__init__.py)']. 
  monkey.patch_all()
Failed to get k8s token: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'
AwsEksResourceDetector failed: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'
AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS.
AwsEc2ResourceDetector failed: <urlopen error [Errno 65] No route to host>
[2025-10-29 11:00:55 -0700] [54080] [INFO] Starting gunicorn 23.0.0
[2025-10-29 11:00:55 -0700] [54080] [INFO] Listening at: http://0.0.0.0:5001 (54080)
[2025-10-29 11:00:55 -0700] [54080] [INFO] Using worker: gevent
[2025-10-29 11:00:55 -0700] [54290] [INFO] Booting worker with pid: 54290
2025-10-29 11:02:12,464 - app - ERROR - Exception on /requests [GET]
```

##### Running with the fix
```shell
((venv) ) ➜  flask git:(fix_gevent_regression) ✗ opentelemetry-instrument gunicorn -w 1 -k gevent --bind 0.0.0.0:5001 app:app
Failed to get k8s token: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'
AwsEksResourceDetector failed: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'
AwsEcsResourceDetector failed: Missing ECS_CONTAINER_METADATA_URI therefore process is not on ECS.
AwsEc2ResourceDetector failed: <urlopen error [Errno 65] No route to host>
[2025-10-29 11:04:45 -0700] [60707] [INFO] Starting gunicorn 23.0.0
[2025-10-29 11:04:45 -0700] [60707] [INFO] Listening at: http://0.0.0.0:5001 (60707)
[2025-10-29 11:04:45 -0700] [60707] [INFO] Using worker: gevent
[2025-10-29 11:04:45 -0700] [60734] [INFO] Booting worker with pid: 60734

```
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

---------

Co-authored-by: Thomas Pierce <[email protected]>

Author: srprash

CHANGELOG.md

@@ -17,3 +17,5 @@ If your change does not need a CHANGELOG entry, add the "skip changelog" label t
     ([#497](https://github.com/aws-observability/aws-otel-python-instrumentation/pull/497))
 - Fix timeout handling for exceeded deadline in retry logic in OTLPAwsLogsExporter
   ([#501](https://github.com/aws-observability/aws-otel-python-instrumentation/pull/501))
+- Fix Gevent patch regression with correct import order
+  ([#522](https://github.com/aws-observability/aws-otel-python-instrumentation/pull/522))

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/aws_opentelemetry_distro.py

@@ -1,5 +1,16 @@
 # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 # SPDX-License-Identifier: Apache-2.0
+
+# flake8: noqa: E402
+# pylint: disable=wrong-import-position
+# ========================================================================
+# Apply the Gevent's patching as the very first step in the distro.
+# IMPORTANT: Do not put any imports before the following 2 lines.
+# Read the comments in the _gevent_patches.py for details.
+from amazon.opentelemetry.distro.patches._gevent_patches import apply_gevent_monkey_patch
+
+apply_gevent_monkey_patch()
+# ========================================================================
 import importlib
 import os
 import sys

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/patches/_gevent_patches.py

@@ -0,0 +1,82 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+from importlib.metadata import PackageNotFoundError, version
+from logging import Logger, getLogger
+
+from packaging.requirements import Requirement
+
+_logger: Logger = getLogger(__name__)
+
+# Env variable to control Gevent monkey patching behavior in ADOT.
+# Read more about the Gevent monkey patching: https://www.gevent.org/intro.html#monkey-patching
+# Possible values are 'all', 'none', and
+# comma separated list 'os, thread, time, sys, socket, select, ssl, subprocess, builtins, signal, queue, contextvars'.
+# When set to 'none', gevent's monkey patching is skipped.
+# When set to 'all' (default behavior), gevent patch is executed for all modules as per
+# https://www.gevent.org/api/gevent.monkey.html#gevent.monkey.patch_all.
+# When set to a comma separated list of modules, only those are processed for gevent's patch.
+AWS_GEVENT_PATCH_MODULES = "AWS_GEVENT_PATCH_MODULES"
+
+
+def _is_gevent_installed() -> bool:
+    """Is the gevent package installed?"""
+    req = Requirement("gevent")
+    try:
+        dist_version = version(req.name)
+        _logger.debug("Gevent is installed: %s", dist_version)
+    except PackageNotFoundError as exc:
+        _logger.debug("Gevent is not installed. %s", exc)
+        return False
+    return True
+
+
+def apply_gevent_monkey_patch():
+    # This patch differs from other instrumentation patches in this directory as it addresses
+    # application compatibility rather than telemetry functionality. It prevents breaking user
+    # applications that run on Gevent and use libraries like boto3, requests, or urllib3 when
+    # instrumented with ADOT.
+    #
+    # Without this patch, users encounter "RecursionError: maximum recursion depth exceeded"
+    # because by the time Gevent monkey-patches modules (such as ssl), those modules have already
+    # been imported by ADOT. Specifically, aws_xray_remote_sampler imports requests, which
+    # transitively imports ssl, leaving these modules in an inconsistent state for Gevent.
+    #
+    # Gevent recommends monkey-patching as early as possible:
+    # https://www.gevent.org/intro.html#monkey-patching
+    #
+    # Since ADOT initialization occurs before user application code, we perform the monkey-patch
+    # here to ensure proper module state for Gevent-based applications.
+
+    # Only apply the gevent monkey patch if gevent is installed is user application space.
+    if _is_gevent_installed():
+        try:
+            gevent_patch_module = os.environ.get(AWS_GEVENT_PATCH_MODULES, "all")
+
+            if gevent_patch_module != "none":
+                # pylint: disable=import-outside-toplevel
+                # Delay import to only occur if monkey patch is needed (e.g. gevent is used to run application).
+                from gevent import monkey
+
+                if gevent_patch_module == "all":
+                    monkey.patch_all()
+                else:
+                    module_list = [module.strip() for module in gevent_patch_module.split(",")]
+
+                    monkey.patch_all(
+                        socket="socket" in module_list,
+                        time="time" in module_list,
+                        select="select" in module_list,
+                        thread="thread" in module_list,
+                        os="os" in module_list,
+                        ssl="ssl" in module_list,
+                        subprocess="subprocess" in module_list,
+                        sys="sys" in module_list,
+                        builtins="builtins" in module_list,
+                        signal="signal" in module_list,
+                        queue="queue" in module_list,
+                        contextvars="contextvars" in module_list,
+                    )
+        except Exception as exc:  # pylint: disable=broad-except
+            _logger.error("Failed to monkey patch gevent, exception: %s", exc)