New Example: examples/hybrid_setup (#246)

Author: Nusnus

.github/workflows/examples.yml

@@ -61,7 +61,7 @@ jobs:
         working-directory: examples/myworker
         timeout-minutes: 10
         run: |
-          pytest -vv tests
+          pytest -xsv tests
 
   range:
     runs-on: ${{ matrix.os }}
@@ -94,7 +94,7 @@ jobs:
         working-directory: examples/range
         timeout-minutes: 30
         run: |
-          pytest -vv tests
+          pytest -xsv tests
 
   rabbitmq_management:
     runs-on: ${{ matrix.os }}
@@ -127,7 +127,7 @@ jobs:
         working-directory: examples/rabbitmq_management
         timeout-minutes: 10
         run: |
-          pytest -vv tests
+          pytest -xsv tests
 
   django:
     runs-on: ${{ matrix.os }}
@@ -166,7 +166,7 @@ jobs:
         timeout-minutes: 10
         run: |
           export DJANGO_SETTINGS_MODULE=proj.settings
-          pytest -vv tests
+          pytest -xsv tests
 
   myutils:
     runs-on: ${{ matrix.os }}
@@ -199,7 +199,7 @@ jobs:
         working-directory: examples/myutils
         timeout-minutes: 10
         run: |
-          pytest -vv tests
+          pytest -xsv tests
 
   worker_pool:
     runs-on: ${{ matrix.os }}
@@ -232,4 +232,37 @@ jobs:
         working-directory: examples/worker_pool
         timeout-minutes: 10
         run: |
-          pytest -vv tests
+          pytest -xsv tests
+
+  hybrid_setup:
+    runs-on: ${{ matrix.os }}
+
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.12"]
+        os: ["ubuntu-latest"]
+
+    steps:
+      - name: Install apt packages
+        if: startsWith(matrix.os, 'ubuntu-')
+        run: |
+          sudo apt update
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+          cache-dependency-path: '**/setup.py'
+      - name: Install dependencies
+        working-directory: examples/hybrid_setup
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Run tests
+        working-directory: examples/hybrid_setup
+        timeout-minutes: 10
+        run: |
+          pytest -xsv tests

docs/userguide/examples/hybrid_setup.rst

@@ -0,0 +1,230 @@
+.. _examples_hybrid_setup:
+
+==============
+ hybrid_setup
+==============
+
+:Release: |version|
+:Date: |today|
+
+.. contents::
+    :local:
+    :depth: 2
+
+Description
+===========
+
+The purpose of this example is to demonstrate a more complex setup with multiple components.
+The example is using two brokers, with the failover feature, a backend and multiple workers of different pools and versions.
+One of the workers is using gevent with the latest Celery release on the default queue,
+while the other is using prefork with Celery 4 and its own queue.
+
+It uses the following workflow to utilize both workers:
+
+.. code-block:: python
+
+    canvas = (
+        group(
+            identity.si("Hello, "),
+            identity.si("world!"),
+        )
+        | noop.s().set(queue="legacy")
+        | identity.si("Done!")
+    )
+
+Highlights
+~~~~~~~~~~
+
+1. No default components.
+2. Session broker and backend components.
+    - Shared between tests, but not between :pypi:`pytest-xdist` sessions.
+    - Only the workers are created again for each test case.
+3. Injects tasks and signal handlers modules to all workers.
+
+This example is based on,
+
+- The :ref:`examples_myworker` example.
+- The :ref:`examples_worker_pool` example.
+
+Breakdown
+=========
+
+File Structure
+~~~~~~~~~~~~~~
+
+The following diagram lists the relevant files in the project.
+
+.. code-block:: text
+
+    hybrid_setup/
+    ├── requirements.txt
+    └── tests/
+        ├── conftest.py
+        ├── test_hybrid_setup.py
+        └── vendors/
+            ├── __init__.py
+            ├── memcached.py
+            ├── rabbitmq.py
+            └── workers/
+                ├── __init__.py
+                ├── gevent.Dockerfile
+                ├── gevent.py
+                ├── legacy.Dockerfile
+                ├── legacy.py
+                ├── signals.py
+                └── tasks.py
+
+requirements.txt
+~~~~~~~~~~~~~~~~
+
+Take a look at the requirements file for this example:
+
+.. literalinclude:: ../../../examples/hybrid_setup/requirements.txt
+    :language: text
+    :caption: examples.hybrid_setup.requirements.txt
+
+Take note the :pypi:`gevent` can be installed independently from the :pypi:`celery` package.
+
+conftest.py
+~~~~~~~~~~~
+
+The ``conftest.py`` file will be used to aggregate each individual configuration. To understand how it works,
+we'll split the file into three parts.
+
+1. Creating the docker network for the components.
+2. Configuring the broker, backend and workers for the setup.
+3. Injecting the tasks and signal handlers modules.
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/conftest.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.conftest.py
+    :start-after: # ----------------------------
+
+test_hybrid_setup.py
+~~~~~~~~~~~~~~~~~~~~
+
+Every test case that uses the :func:`celery_setup <pytest_celery.fixtures.setup.celery_setup>` fixture will run
+its scenario on the setup that was configured in the ``conftest.py`` file.
+
+For this example, we have the following test cases.
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/test_hybrid_setup.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.test_hybrid_setup.py
+    :start-after: TestHybridSetupExample
+
+.. tip::
+
+    The components themselves can be used in the test case to easily access their attributes and methods, like shown
+    in the failover test case. When used without the :func:`celery_setup <pytest_celery.fixtures.setup.celery_setup>`
+    fixture, the components will run independently and might not be aware of each other.
+
+rabbitmq.py and memcached.py
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The brokers and result backend are defined as independent components that are being configured into the setup
+using the ``conftest.py`` file. They add **session scope** fixtures and integrate using the matching :ref:`node class <test-nodes>`.
+
+Main | Failover Brokers
+-----------------------
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/rabbitmq.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.rabbitmq.py
+
+Result Backend
+--------------
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/memcached.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.memcached.py
+
+gevent.py and gevent.Dockerfile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These files are taken from the :ref:`test_gevent_pool` example with one simple change.
+
+.. code-block:: python
+
+    RUN pip install "celery[gevent]" "pytest-celery[all]==1.0.0b4"
+
+The Dockerfile doesn't use the requirements file, but instead installs the packages directly.
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/gevent.Dockerfile
+    :language: docker
+    :caption: examples.hybrid_setup.tests.vendors.workers.gevent.Dockerfile
+
+.. note::
+
+    The :ref:`test_gevent_pool` example defines everything in the test file. Here we use the ``gevent.py`` file.
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/gevent.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.workers.gevent.py
+
+legacy.py and legacy.Dockerfile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The "legacy" worker is basically Celery 4 worker with the prefork pool. Very similar to the gevent worker,
+we add a new Dockerfile and worker module.
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/legacy.Dockerfile
+    :language: docker
+    :caption: examples.hybrid_setup.tests.vendors.workers.legacy.Dockerfile
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/legacy.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.workers.legacy.py
+
+.. tip::
+
+    Check all of the configurations above again and notice the usage of ``hybrid_setup_example_network``.
+    See how both session and non-session fixtures are sharing the same session docker network.
+
+tasks.py and signals.py
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The tasks and signal handlers are being injected into the workers using the ``conftest.py`` file,
+according to the documentation:
+
+1. :ref:`injecting-tasks`.
+2. :ref:`injecting-signals-handlers`.
+
+The files themselves are very simple,
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/tasks.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.workers.tasks.py
+
+.. literalinclude:: ../../../examples/hybrid_setup/tests/vendors/workers/signals.py
+    :language: python
+    :caption: examples.hybrid_setup.tests.vendors.workers.signals.py
+
+And again, from ``confest.py``,
+
+.. code-block:: python
+
+    @pytest.fixture
+    def default_worker_tasks(default_worker_tasks: set) -> set:
+        from tests.vendors.workers import tasks
+
+        default_worker_tasks.add(tasks)
+        return default_worker_tasks
+
+
+    @pytest.fixture
+    def default_worker_signals(default_worker_signals: set) -> set:
+        from tests.vendors.workers import signals
+
+        default_worker_signals.add(signals)
+        return default_worker_signals
+
+.. note::
+
+    The tasks and signals are being injected into the workers that use the default volume with:
+
+    .. code-block:: python
+
+        volumes={"{default_worker_volume.name}": defaults.DEFAULT_WORKER_VOLUME},
+
+    Both our workers are using the default volume, so we only need to inject the tasks and signals once.

docs/userguide/examples/index.rst

@@ -22,3 +22,4 @@ Every example is an independent project and is tested via the
     range
     myutils
     django
+    hybrid_setup

docs/userguide/examples/worker_pool.rst

@@ -53,7 +53,7 @@ The purpose of this worker is to ensure the gevent dependency is installed.
    :caption: examples.worker_pool.Dockerfile
 
 .. literalinclude:: ../../../examples/worker_pool/requirements.txt
-   :language: docker
+   :language: text
    :caption: examples.worker_pool.requirements.txt
 
 tasks.py
@@ -65,6 +65,8 @@ Our tasks module is using the example task from the `Celery gevent example <http
    :language: python
    :caption: examples.worker_pool.tasks.py
 
+.. _test_gevent_pool:
+
 test_gevent_pool.py
 ~~~~~~~~~~~~~~~~~~~
 

examples/hybrid_setup/pytest.ini

@@ -0,0 +1,5 @@
+[pytest]
+log_cli = true
+log_cli_level = INFO
+log_cli_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)
+log_cli_date_format = %Y-%m-%d %H:%M:%S

examples/hybrid_setup/requirements.txt

@@ -0,0 +1,5 @@
+pytest>=7.4.4
+pytest-xdist>=3.5.0
+pytest-subtests>=0.11.0
+celery[gevent]
+pytest-celery[all]@git+https://github.com/celery/pytest-celery.git