Add Httpx plugin (#283)

Author: XinweiLyu

CHANGELOG.md

@@ -26,6 +26,7 @@
   - Add HBase plugin Python HappyBase model  (#266) 
   - Add FastAPI plugin websocket protocol support (#269)
   - Add Websockets (client) plugin (#269)
+  - Add HTTPX plugin (#283)
 
 - Fixes:
   - Allow RabbitMQ BlockingChannel.basic_consume() to link with outgoing spans (#224)

docs/en/contribution/Developer.md

@@ -25,6 +25,8 @@ We have migrated from basic pip to [Poetry](https://python-poetry.org/) to manag
 Once you have `make` ready, run `make env`, this will automatically install the right Poetry release, and create 
 (plus manage) a `.venv` virtual environment for us based on the currently activated Python 3 version. Enjoy coding!
 
+Note: Make sure you have `python3` aliased to `python` available on Windows computers instead of pointing to the Microsoft app store.
+
 ### Switching between Multiple Python Versions
 Do not develop/test on Python < 3.7, since Poetry and some other functionalities we implement rely on Python 3.7+
 

docs/en/contribution/How-to-develop-plugin.md

@@ -3,24 +3,25 @@
 You can always take [the existing plugins](../setup/Plugins.md) as examples, while there are some general ideas for all plugins.
 1. A plugin is a module under the directory `skywalking/plugins` with an `install` method; 
 2. Inside the `install` method, you find out the relevant method(s) of the libraries that you plan to instrument, and create/close spans before/after those method(s).
-3. You should also provide version rules in the plugin module, which means the version of package your plugin support. You should init a dict with keys `name` and `rules`. the `name` is your plugin's corresponding package's name, the `rules` is the version rules this package should follow.
-   
-   You can use >, >=, ==, <=, <, and != operators in rules. 
-   
-   The relation between rules element in the rules array is **OR**, which means the version of the package should follow at least one rule in rules array.
-   
-   You can set many version rules in one element of rules array, separate each other with a space character, the relation of rules in one rule element is **AND**, which means the version of package should follow all rules in this rule element.
-   
-   For example, below `version_rule` indicates that the package version of `django` should `>=2.0 AND <=2.3 AND !=2.2.1` OR `>3.0`.
+3. You should also provide version rules in the plugin module, which means the version of package your plugin aim to test. 
+
+   All below variables will be used by the tools/plugin_doc_gen.py to produce a latest [Plugin Doc](../setup/Plugins.md).
+
    ```python
-   version_rule = {
-       "name": "django",
-       "rules": [">=2.0 <=2.3 !=2.2.1", ">3.0"]
+   link_vector = ['https://www.python-httpx.org/']  # This should link to the official website/doc of this lib
+   # The support matrix is for scenarios where some libraries don't work for certain Python versions
+   # Therefore, we use the matrix to instruct the CI testing pipeline to skip over plugin test for such Python version
+   # The right side versions, should almost always use A.B.* to test the latest minor version of two recent major versions. 
+   support_matrix = {
+       'httpx': {
+           '>=3.7': ['0.23.*', '0.22.*']
+       }
    }
+   # The note will be used when generating the plugin documentation for users.
+   note = """"""
    ```
 4. Every plugin requires a corresponding test under `tests/plugin` before it can be merged, refer to the [Plugin Test Guide](How-to-test-plugin.md) when writing a plugin test.
-5. Update the [Supported Plugin List](../setup/Plugins.md).
-6. Add the environment variables to [Environment Variable list](../setup/Configuration.md) if any.
+5. Add the corresponding configuration options added/modified by the new plugin to the config.py and add new comments for each, then regenerate the `configuration.md` by `make doc-gen`.
 
 ## Steps after coding
 

docs/en/contribution/How-to-test-locally.md

@@ -11,6 +11,8 @@ Please first refer to the [Developer Guide](Developer.md) to set up a developmen
 
 TL;DR: run ``make env``. This will create virtual environments for python and generate the protocol folder needed for the agent.
 
+Note: Make sure you have `python3` aliased to `python` available on Windows computers instead of pointing to the Microsoft app store.
+
 By now, you can do what you want. Let's get to the topic of how to test.
 
 The test process requires `docker` and `docker-compose` throughout. If you haven't installed them, please install them first.

docs/en/setup/Plugins.md

@@ -29,6 +29,7 @@ or a limitation of SkyWalking auto-instrumentation (welcome to contribute!)
 | [happybase](https://happybase.readthedocs.io) | Python >=3.7 - ['1.2.0'];  | `sw_happybase` |
 | [http_server](https://docs.python.org/3/library/http.server.html) | Python >=3.7 - ['*'];  | `sw_http_server` |
 | [werkzeug](https://werkzeug.palletsprojects.com/) | Python >=3.7 - ['1.0.1', '2.0'];  | `sw_http_server` |
+| [httpx](https://www.python-httpx.org/) | Python >=3.7 - ['0.23.*', '0.22.*'];  | `sw_httpx` |
 | [kafka-python](https://kafka-python.readthedocs.io) | Python >=3.7 - ['2.0'];  | `sw_kafka` |
 | [loguru](https://pypi.org/project/loguru/) | Python >=3.7 - ['0.6.0'];  | `sw_loguru` |
 | [mysqlclient](https://mysqlclient.readthedocs.io/) | Python >=3.7 - ['2.1.*'];  | `sw_mysqlclient` |

poetry.lock

@@ -127,6 +127,7 @@ asyncpg = "^0.27.0"
 happybase = "1.2.0"
 websockets = "^10.4"
 loguru = "^0.6.0"
+httpx = "^0.23.3"
 
 [tool.poetry.group.lint.dependencies]
 flake8 = "^5.0.4"

pyproject.toml

@@ -51,6 +51,7 @@ class Component(Enum):
     AsyncPG = 7016
     AIORedis = 7017
     Websockets = 7018
+    HTTPX = 7019
 
 
 class Layer(Enum):

skywalking/__init__.py

@@ -0,0 +1,102 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from skywalking import Layer, Component, config
+from skywalking.trace.context import get_context, NoopContext
+from skywalking.trace.span import NoopSpan
+from skywalking.trace.tags import TagHttpMethod, TagHttpURL, TagHttpStatusCode
+
+link_vector = ['https://www.python-httpx.org/']
+support_matrix = {
+    'httpx': {
+        '>=3.7': ['0.23.*', '0.22.*']
+    }
+}
+note = """"""
+
+
+def install():
+    from httpx import _client
+    from httpx import USE_CLIENT_DEFAULT
+
+    _async_send = _client.AsyncClient.send
+    _send = _client.Client.send
+
+    async def _sw_async_send(self, request, *, stream=False, auth=USE_CLIENT_DEFAULT,
+                             follow_redirects=USE_CLIENT_DEFAULT, ):
+        url_object = request.url
+
+        span = NoopSpan(NoopContext()) if config.ignore_http_method_check(
+            request.method) else get_context().new_exit_span(op=url_object.path or '/', peer=url_object.netloc.decode(),
+                                                             component=Component.HTTPX)
+        with span:
+            carrier = span.inject()
+            span.layer = Layer.Http
+
+            if not request.headers:
+                request.headers = {}
+            for item in carrier:
+                request.headers[item.key] = item.val
+
+            span.tag(TagHttpMethod(request.method.upper()))
+            url_safe = str(url_object).replace(url_object.username, '').replace(url_object.password, '')
+
+            span.tag(TagHttpURL(url_safe))
+
+            res = await _async_send(self, request, stream=stream, auth=auth, follow_redirects=follow_redirects)
+
+            status_code = res.status_code
+            span.tag(TagHttpStatusCode(status_code))
+
+            if status_code >= 400:
+                span.error_occurred = True
+
+            return res
+
+    _client.AsyncClient.send = _sw_async_send
+
+    def _sw_send(self, request, *, stream=False, auth=USE_CLIENT_DEFAULT, follow_redirects=USE_CLIENT_DEFAULT, ):
+        url_object = request.url
+
+        span = NoopSpan(NoopContext()) if config.ignore_http_method_check(
+            request.method) else get_context().new_exit_span(op=url_object.path or '/', peer=url_object.netloc.decode(),
+                                                             component=Component.HTTPX)
+        with span:
+            carrier = span.inject()
+            span.layer = Layer.Http
+
+            if not request.headers:
+                request.headers = {}
+            for item in carrier:
+                request.headers[item.key] = item.val
+
+            span.tag(TagHttpMethod(request.method.upper()))
+            url_safe = str(url_object).replace(url_object.username, '').replace(url_object.password, '')
+
+            span.tag(TagHttpURL(url_safe))
+
+            res = _send(self, request, stream=stream, auth=auth, follow_redirects=follow_redirects)
+
+            status_code = res.status_code
+            span.tag(TagHttpStatusCode(status_code))
+
+            if status_code >= 400:
+                span.error_occurred = True
+
+            return res
+
+    _client.Client.send = _sw_send