Add prefork support for uwsgi and gunicorn (#288)

* Add prefork support for uwsgi and gunicorn

Author: Superskyyy

.github/workflows/CI.yaml

@@ -103,13 +103,12 @@ jobs:
     name: Build Plugin and UT Matrix
     needs: [ license-and-lint, changes, plugin-doc-check ]
     if: |
-      ( always() && ! cancelled() ) && 
+      ( always() && ! cancelled() ) &&
       ((github.event_name == 'schedule' && github.repository == 'apache/skywalking-python') || needs.changes.outputs.agent == 'true')
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
       - id: set-matrix
-        # TODO change to github output
         run: |
           sudo apt-get install jq
           echo "matrix=$(bash tests/gather_test_paths.sh)" >> $GITHUB_OUTPUT
@@ -158,7 +157,7 @@ jobs:
     timeout-minutes: 20
     strategy:
       matrix:
-        python-version: [ "3.7", "3.8", "3.9", "3.10", "3.11" ]  
+        python-version: [ "3.7", "3.8", "3.9", "3.10", "3.11" ]
         test-path: ${{fromJson(needs.prep-plugin-and-unit-tests.outputs.matrix)}}
       fail-fast: false
     env:
@@ -229,12 +228,24 @@ jobs:
       matrix:
         python-image-variant: [ "3.7-slim", "3.8-slim", "3.9-slim", "3.10-slim", "3.11-slim" ]
         case:
-          - name: gRPC
-            path: tests/e2e/case/grpc/e2e.yaml
-          - name: HTTP
-            path: tests/e2e/case/http/e2e.yaml
-          - name: Kafka
-            path: tests/e2e/case/kafka/e2e.yaml
+          - name: gRPC-single-process
+            path: tests/e2e/case/grpc/single/e2e.yaml
+          - name: gRPC-gunicorn
+            path: tests/e2e/case/grpc/gunicorn/e2e.yaml
+          - name: gRPC-uwsgi
+            path: tests/e2e/case/grpc/uwsgi/e2e.yaml
+          - name: HTTP-single-process
+            path: tests/e2e/case/http/single/e2e.yaml
+          - name: HTTP-gunicorn
+            path: tests/e2e/case/http/gunicorn/e2e.yaml
+          - name: HTTP-uwsgi
+            path: tests/e2e/case/http/uwsgi/e2e.yaml
+          - name: Kafka-single-process
+            path: tests/e2e/case/kafka/single/e2e.yaml
+          - name: Kafka-gunicorn
+            path: tests/e2e/case/kafka/gunicorn/e2e.yaml
+          - name: Kafka-uwsgi
+            path: tests/e2e/case/kafka/uwsgi/e2e.yaml
           - name: profiling_threading
             path: tests/e2e/case/profiling/threading/e2e.yaml
           - name: profiling_greenlet
@@ -270,7 +281,7 @@ jobs:
     timeout-minutes: 60
     # wait upon them regardless of success or skipped or failure
     if: ${{ always() }}
-    needs: [ license-and-lint, changes, plugin-and-unit-tests, plugin-doc-check, e2e-tests ] 
+    needs: [ license-and-lint, changes, plugin-and-unit-tests, plugin-doc-check, e2e-tests ]
     steps:
       - name: Merge Requirement
         # check license, lint, plugin and e2e tests, then naturally exits 0

CHANGELOG.md

@@ -8,7 +8,8 @@
   please check with the latest official documentation before upgrading. (#273, #282)
   https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
   - **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
-
+  - **BREAKING**: DockerHub Python agent images since v1.0.0 will no longer include the `run` part in `ENTRYPOINT ["sw-python", "run"]`, 
+  user should prefix their command with `[-d/--debug] run [-p/--prefork] <Command>` for extra flexibility.
 
 - Feature:
   - Add support for Python 3.11 (#285)
@@ -19,7 +20,8 @@
   - Add support for the tags of Virtual Cache for Redis (#263)
   - Add a new configuration `kafka_namespace` to prefix the kafka topic names (#277)
   - Add log reporter support for loguru (#276)
-  - Add **experimental** support for explicit os.fork(), restarts agent in new process (#286)
+  - Add **experimental** support for explicit os.fork(), restarts agent in forked process (#286)
+  - Add **experimental** sw-python CLI `sw-python run [-p]` flag (-p/--prefork) to enable non-intrusive uWSGI and Gunicorn postfork support (#288)
 
 - Plugins:
   - Add aioredis, aiormq, amqp, asyncpg, aio-pika, kombu RMQ plugins (#230 Missing test coverage) 

demo/docker-compose.yaml

@@ -65,7 +65,7 @@ services:
       ZOOKEEPER_TICK_TIME: 2000
     networks:
       - manual
-
+  
   kafka:
     container_name: kafka
     image: confluentinc/cp-kafka
@@ -86,21 +86,21 @@ services:
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
     networks:
       - manual
-
-#
-#  kafka-ui:
-#    image: provectuslabs/kafka-ui
-#    container_name: kafka-ui
-#    ports:
-#      - "8088:8080"
-#    restart: always
-#    environment:
-#      - KAFKA_CLUSTERS_0_NAME=local
-#      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
-#    depends_on:
-#      - kafka
-#    networks:
-#      - manual
+  
+  
+  kafka-ui:
+    image: provectuslabs/kafka-ui
+    container_name: kafka-ui
+    ports:
+      - "8088:8080"
+    restart: always
+    environment:
+      - KAFKA_CLUSTERS_0_NAME=local
+      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
+    depends_on:
+      - kafka
+    networks:
+      - manual
 
 networks:
   manual:

demo/flask_consumer_prefork.py

@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+sw-python -d run -p uwsgi --die-on-term \
+    --http 0.0.0.0:9090 \
+    --http-manage-expect \
+    --workers 2 \
+    --worker-reload-mercy 30 \
+    --enable-threads \
+    --threads 1 \
+    --manage-script-name \
+    --mount /=flask_consumer_prefork:app
+"""
+from flask import Flask
+import logging
+
+app = Flask(__name__)
+
+
+# This is wrong!! Do not do this with prefork server, fork support (os.fork) do not work with uWSGI!
+# from skywalking import agent, config
+#
+# config.init(agent_collector_backend_services='127.0.0.1:11800', agent_name='your awesome service',
+# agent_logging_level ='DEBUG', agent_experimental_fork_support=True)
+#
+# agent.start()
+
+
+@app.route('/cat', methods=['POST', 'GET'])
+def artist():
+    try:
+        logging.critical('fun cat got a request')
+        return {'Cat Fun Fact': 'Fact is cat, cat is fat'}
+    except Exception as e:  # noqa
+        return {'message': str(e)}
+
+
+if __name__ == '__main__':
+    # noinspection PyTypeChecker
+    # app.run(host='0.0.0.0', port=9090)
+    ...

demo/gunicorn_consumer_prefork.py

@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+This one demos how to use gunicorn with a simple fastapi uvicorn app.
+"""
+import logging
+
+from fastapi import FastAPI
+
+"""
+# Run this to see sw-python working with gunicorn
+sw-python -d run -p \
+    gunicorn gunicorn_consumer_prefork:app \
+    --workers 2 --worker-class uvicorn.workers.UvicornWorker \
+    --bind 0.0.0.0:8088
+"""
+
+app = FastAPI()
+
+
+@app.get('/cat')
+async def application():
+    try:
+        logging.critical('fun cat got a request')
+        return {'Cat Fun Fact': 'Fact is cat, cat is fat'}
+    except Exception as e:  # noqa
+        return {'message': str(e)}
+
+
+if __name__ == '__main__':
+    # noinspection PyTypeChecker
+    # uvicorn.run(app, host='0.0.0.0', port=8088)
+    ...

docker/Dockerfile

@@ -24,4 +24,4 @@ ARG SW_PYTHON_AGENT_VERSION
 RUN pip install --no-cache-dir "apache-skywalking[${SW_PYTHON_AGENT_PROTOCOL}]==${SW_PYTHON_AGENT_VERSION}"
 
 # So that the agent can be auto-started when container is started
-ENTRYPOINT ["sw-python", "run"]
+ENTRYPOINT ["sw-python"]

docs/en/setup/CLI.md

@@ -1,8 +1,11 @@
-# SkyWalking Python Agent Command-Line Interface (sw-python CLI)
+# SkyWalking Python Agent Command Line Interface (sw-python CLI)
 
-In releases before 0.7.0, you would at least need to add the following lines to your applications to get the agent attached and running.  
+**Now, SkyWalking Python Agent CLI is the recommended way of running your application with Python agent, 
+the CLI is well-tested and used by all agent E2E & Plugin tests.**
 
-This is the recommended way of running your application with Python agent.
+
+In releases before 0.7.0, you would at least need to add the following lines to your applications to get the agent attached and running, 
+this can be tedious in many cases due to large number of services, DevOps practices and can cause problem when used with prefork servers.
 
 ```python
 from skywalking import agent, config
@@ -11,56 +14,65 @@ agent.start()
 ```
 
 
-Now the SkyWalking Python agent implements a command-line interface that can be utilized to attach the agent to your
+The SkyWalking Python agent implements a command-line interface that can be utilized to attach the agent to your
 awesome applications during deployment **without changing any application code**, 
 just like the [SkyWalking Java Agent](https://github.com/apache/skywalking-java).
 
+> The following feature is added in v1.0.0 as experimental flag, so you need to specify the -p flag to `sw-python run -p`. 
+> In the future, this flag will be removed and agent will automatically enable prefork/fork support in a more comprehensive manner.
+
+Especially with the new automatic postfork injection feature, you no longer have to worry about threading and forking incompatibility.
+
+Check [How to use with uWSGI](faq/How-to-use-with-uwsgi.md) and [How to use with Gunicorn](faq/How-to-use-with-gunicorn.md) to understand
+the detailed background on what is post_fork, why you need them and how to easily overcome the trouble with `sw-python` CLI.
+
+You should still read the [legacy way](Intrusive.md) to integrate agent in case the `sw-python` CLI is not working for you.
+
+
+
 ## Usage
 
 Upon successful [installation of the SkyWalking Python agent via pip](Installation.md#from-pypi),
 a command-line script `sw-python` is installed in your environment (virtual env preferred).
 
-run `sw-python` to see if it is available.
+> run `sw-python` to see if it is available, you will need to pass configuration by environment variables.
+
+For example: `export SW_AGENT_COLLECTOR_BACKEND_SERVICES=localhost:11800`
 
 ### The `run` option
 
-Currently, the `sw-python` CLI provides a `run` option, which you can use to execute your applications
+The `sw-python` CLI provides a `run` option, which you can use to execute your applications
 (either begins with the `python` command or Python-based programs like `gunicorn` on your path) 
 just like you invoke them normally, plus a prefix, the following example demonstrates the usage.
 
-If your previous command to run your gunicorn application is:
+If your previous command to run your gunicorn/uwsgi application is:
 
-`gunicorn app.wsgi`
+`gunicorn your_app:app --workers 2 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8088`
 
-Please change it to:
+or
 
-`sw-python run gunicorn app.wsgi`
+`uwsgi --die-on-term --http 0.0.0.0:5000 --http-manage-expect --master --workers 3 --enable-threads --threads 3 --manage-script-name --mount /=main:app`
 
-The SkyWalking Python agent will startup along with your application shortly.
+Please change it to (**the `-p` option starts one agent in each process, which is the correct behavior**):
 
-Note that the command does work with multiprocessing and subprocess as long as the `PYTHONPATH` is inherited, 
-please configure the environment variables configuration accordingly based on the general documentation.
+**Important:** if the call to uwsgi/gunicorn is prefixed with other commands, this approach will fail 
+since agent currently looks for the command line input at index 0 for safety as an experimental feature.
 
-When executing commands with `sw-python run command`, your command's Python interpreter will pick up the SkyWalking loader module.
+`sw-python run -p gunicorn your_app:app --workers 2 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8088`
 
-It is not safe to attach SkyWalking Agent to those commands that resides in another Python installation 
-because incompatible Python versions and mismatched SkyWalking versions can cause problems. 
-Therefore, any attempt to pass a command that uses a different Python interpreter/ environment will not bring up 
-SkyWalking Python Agent even if another SkyWalking Python agent is installed there(no matter the version), 
-and will force exit with an error message indicating the reasoning.
+or 
 
-#### Disabling child processes from starting new agents
+`sw-python run -p uwsgi --die-on-term --http 0.0.0.0:5000 --http-manage-expect --master --workers 3 --enable-threads --threads 3 --manage-script-name --mount /=main:app`
 
-Sometimes you don't actually need the agent to monitor anything in a child process.
 
-If you do not need the agent to get loaded for application child processes, you can turn off the behavior by setting an environment variable.
+The SkyWalking Python agent will start up along with all your application workers shortly.
 
-`SW_AGENT_SW_PYTHON_BOOTSTRAP_PROPAGATE` to `False`
+Note that `sw-python` also work with spawned subprocess (os.exec*/subprocess) as long as the `PYTHONPATH` is inherited. 
 
-Note the auto bootstrap depends on the environment inherited by child processes, 
-thus prepending a new sitecustomize path to or removing the loader path from the `PYTHONPATH` could prevent the agent from loading in a child process. 
+Additionally, `sw-python` started agent works well with `os.fork` when your application forks workers, 
+as long as the `SW_AGENT_EXPERIMENTAL_FORK_SUPPORT` is turned on. (It will be automatically turned on when gunicorn is detected)
 
-### Configuring the agent 
+## Configuring the agent 
 
 You would normally want to provide additional configurations other than the default ones.
 
@@ -69,13 +81,13 @@ You would normally want to provide additional configurations other than the defa
 The currently supported method is to provide the environment variables listed 
 and explained in the [Environment Variables List](Configuration.md).
 
-#### Through a sw-config.yaml (TBD)
+#### Through a sw-config.toml (TBD)
 
-Currently, only environment variable configuration is supported; an optional `yaml` configuration is to be implemented.
+Currently, only environment variable configuration is supported; an optional `toml` configuration is to be implemented.
 
-### Enabling CLI DEBUG mode
+## Enabling CLI DEBUG mode
 
-Note the CLI is a new feature that manipulates the Python interpreter bootstrap behaviour, there could be unsupported cases.
+Note the CLI is a feature that manipulates the Python interpreter bootstrap behaviour, there could be unsupported cases.
 
 If you encounter unexpected problems, please turn on the DEBUG mode by adding the `-d` or `--debug` flag to your `sw-python` command, as shown below.
 
@@ -86,6 +98,29 @@ To: `sw-python -d run command`
 Please attach the debug logs to the [SkyWalking Issues](https://github.com/apache/skywalking/issues) section if you believe it is a bug,
 [idea discussions](https://github.com/apache/skywalking/discussions) and [pull requests](https://github.com/apache/skywalking-python/pulls) are always welcomed.
 
+
+## Additional Remarks
+
+When executing commands with `sw-python run command`, your command's Python interpreter will pick up the SkyWalking loader module.
+
+It is not safe to attach SkyWalking Agent to those commands that resides in another Python installation 
+because incompatible Python versions and mismatched SkyWalking versions can cause problems. 
+Therefore, any attempt to pass a command that uses a different Python interpreter/ environment will not bring up 
+SkyWalking Python Agent even if another SkyWalking Python agent is installed there(no matter the version), 
+and will force exit with an error message indicating the reasoning.
+
+#### Disabling spawned processes from starting new agents
+
+Sometimes you don't actually need the agent to monitor anything in a new process (when it's not a web service worker). 
+(here we mean process spawned by subprocess and os.exec*(), os.fork() is not controlled by this flag but `experimental_fork_support`)
+
+If you do not need the agent to get loaded for application child processes, you can turn off the behavior by setting an environment variable.
+
+`SW_AGENT_SW_PYTHON_BOOTSTRAP_PROPAGATE` to `False`
+
+Note the auto bootstrap depends on the environment inherited by child processes, 
+thus prepending a new sitecustomize path to or removing the loader path from the `PYTHONPATH` could also prevent the agent from loading in a child process. 
+
 #### Known limitations
 
 1. The CLI may not work properly with arguments that involve double quotation marks in some shells.