Docs and demo housekeeping (#87)

* Make the demo more pretty

* Delete 2 docs file and fix demo

* Remove more we do not need

* Fix usage link

* Review comment

* Implement latest round of review comments

Author: AlanCoding

README.md

@@ -1,36 +1,41 @@
 <!-- License Badge -->
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/ansible/dispatcher/blob/main/LICENSE)
 
-This is intended to be a working space for prototyping a code split of:
-
-<https://github.com/ansible/awx/tree/devel/awx/main/dispatch>
+## Dispatcher
 
-As a part of doing the split, we also want to resolve a number of
-long-standing design and sustainability issues, thus, asyncio.
-
-The philosophy of the dispatcher is to have a limited scope
+The dispatcher is a service to run python tasks in subprocesses,
+designed specifically to work well with pg_notify,
+but intended to be extensible to other message delivery means.
+Its philosophy is to have a limited scope
 as a "local" runner of background tasks, but to be composable
 so that it can be "wrapped" easily to enable clustering and
 distributed task management by apps using it.
 
+> [!WARNING]
+> This project is in initial development. Expect many changes, including name, paths, and CLIs.
+
 Licensed under [Apache Software License 2.0](LICENSE)
 
 ### Usage
 
 You have a postgres server configured and a python project.
 You will use dispatcher to trigger a background task over pg_notify.
-
 Both your *background dispatcher service* and your *task publisher* process must have
-python configured so that your task is importable.
+python configured so that your task is importable. Instructions are broken into 3 steps:
+
+1. **Library** - Configure dispatcher, mark the python methods you will run with it
+2. **Dispatcher service** - Start your background task service, it will start listening
+3. **Publisher** - From some other script, submit tasks to be ran
 
-For more options, see `docs/usage.md`.
+In the "Manual Demo" section, an runnable example of this is given.
 
 #### Library
 
 The dispatcher `@task()` decorator is used to register tasks.
 
-See the `tools/test_methods.py` module.
-This defines a dispatcher task and the pg_notify channel it will be sent over.
+The [tests/data/methods.py](tests/data/methods.py) module defines some
+dispatcher tasks and the pg_notify channels they will be sent over.
+For more `@task` options, see [docs/task_options.md](docs/task_options.md).
 
 ```python
 from dispatcher.publish import task
@@ -49,10 +54,12 @@ from dispatcher.config import setup
 config = {
     "producers": {
         "brokers": {
-            "pg_notify": {"conninfo": "dbname=postgres user=postgres"},
-            "channels": [
-                "test_channel",
-            ],
+            "pg_notify": {
+                "conninfo": "dbname=postgres user=postgres"
+                "channels": [
+                    "test_channel",
+                ],
+            },
         },
     },
     "pool": {"max_workers": 4},
@@ -62,6 +69,8 @@ setup(config)
 
 For more on how to set up and the allowed options in the config,
 see the section [config](docs/config.md) docs.
+The `queue` passed to `@task` needs to match a pg_notify channel in the `config`.
+It is often useful to have different workers listen to different sets of channels.
 
 #### Dispatcher service
 
@@ -83,8 +92,8 @@ run_service()
 ```
 
 Configuration tells how to connect to postgres, and what channel(s) to listen to.
-The demo has this in `dispatcher.yml`, which includes listening to `test_channel`.
-That matches the `@task` in the library.
+
+
 
 #### Publisher
 
@@ -116,38 +125,64 @@ print_hello.apply_async(args=[], kwargs={})
 The difference is that `apply_async` takes both args and kwargs as kwargs themselves,
 and allows for additional configuration parameters to come after those.
 
-As of writing, this only works if you have a Django connection configured.
-You can manually pass configuration info (as in the demo) for non-Django use.
-
 ### Manual Demo
 
+For this demo, the [tests/data/methods.py](tests/data/methods.py) will be used
+in place of a real app. Making those importable is why `PYTHONPATH` must be
+modified in some steps. The config for this demo can be found in the
+[dispatcher.yml](dispatcher.yml) file, which is a default location
+the `dispatcher-standalone` entrypoint looks for.
+
+Initial setup:
+
+```
+pip install -e .[pg_notify]
+make postgres
+```
+
 You need to have 2 terminal tabs open to run this.
 
 ```
 # tab 1
-make postgres
-PYTHONPATH=$PYTHONPATH:tools/ dispatcher-standalone
+PYTHONPATH=$PYTHONPATH:. dispatcher-standalone
 # tab 2
-python tools/write_messages.py
+./run_demo.py
 ```
 
-This will run the dispatcher with schedules, and process a burst of messages
-that give instructions to run tasks.
+This will run the dispatcher with schedules, and process bursts of messages
+that give instructions to run tasks. Tab 2 will contain some responses
+from the dispatcher service. Tab 1 will show a large volume of logs
+related to processing tasks.
 
 ### Running Tests
 
-A structure has been set up for integration tests.
-The word "integration" only means that postgres must be running.
+Most tests (except for tests/unit/) require postgres to be running.
 
 ```
 pip install -r requirements_dev.txt
 make postgres
-py.test tests/
+pytest tests/
 ```
 
-This accomplishes the most basic of starting and shutting down.
-With no tasks submitted, it should record running 0 tasks,
-and with a task submitted, it records running 1 task.
+### Background
+
+This is intended to be a working space for prototyping a code split of:
+
+<https://github.com/ansible/awx/tree/devel/awx/main/dispatch>
+
+As a part of doing the split, we also want to resolve a number of
+long-standing design and sustainability issues, thus, asyncio.
+For a little more background see [docs/design_notes.md](docs/design_notes.md).
+
+There is documentation of the message formats used by the dispatcher
+in [docs/message_formats.md](docs/message_formats.md). Some of these are internal,
+but some messages are what goes over the user-defined brokers (pg_notify).
+You can trigger tasks using your own "publisher" code as an alternative
+to attached methods like `.apply_async`. Doing this requires connecting
+to postges and submitting a pg_notify message with JSON data
+that conforms to the expected format.
+The `./run_demo.py` script shows examples of this, but borrows some
+connection and config utilities to help.
 
 ## Contributing
 

docs/design_notes.md

@@ -1,4 +1,30 @@
-## Reference Designs
+## Design Notes
+
+Many of the specific choices made in the dispatcher design are to enable pg_notify use.
+The advantage of using pg_notify is that you get an extremely simple topology.
+Imagine a web and a task runner service, invoked separately, using postgres.
+
+```mermaid
+flowchart TD
+
+A(web)
+B(task)
+C(postgres)
+
+A-->C
+B-->C
+```
+
+However, pg_notify is not a true queue, and this drives the design of the dispatcher,
+having its main process listening for messages and farming the work out to worker subprocesses.
+
+This helps with pg_notify use, but still doesn't solve all the problems,
+because those problems ultimately need persistent storage.
+Current plan is to offer a "stock" solution in the form of a django-ansible-base app.
+
+https://github.com/ansible/django-ansible-base
+
+End-goal features and design proposals have been moved to the issue queue.
 
 ### AWX dispatcher
 
@@ -12,14 +38,15 @@ https://github.com/ansible/awx/pull/2266
 
 > ...much like the callback receiver implementation in 3.3.0 (on which this code is based), this entry point is a kombu.ConsumerMixin.
 
-### Kombu
+### Kombu (Celery)
 
-Kombu is a sub-package of celery.
+Kombu was used by AWX before its transition to a custom solution. Kombu is a sub-package of celery.
 
 https://github.com/celery/kombu
 
 In messaging module, this has a `Producer` and `Consumer` classes.
-In mixins it has a `ConsumerMixin`, but no methods seem to have made it into AWX dispatch.
+In mixins it has a `ConsumerMixin`. AWX dispatcher has consumer classes,
+but no methods seem to have made it from kombu into AWX dispatch.
 
 This doesn't deal with worker pool management. It does have examples with `Worker` classes.
 These follow a similar contract with `process_task` here.
@@ -55,60 +82,12 @@ In AWX dispatcher, a full queue may put messages into individual worker IPCs.
 This caused bad results, like delaying tasks due to long-running jobs,
 while the pool had many other workers free up in the mean time.
 
-## Alternative Archectures
-
-This are blue-sky ideas, which may not happen anytime soon,
-but they are described to help structure the app today so it can expand
-into these potential future roles.
-
-### Singleton task queue
-
-A major pivot from the AWX dispatcher is that we do not use 1 result queue per worker,
-but a single result queue for all workers, and each meassage includes a worker id.
-
-If you continue this pattern, then we would no longer have a call queue for each worker,
-and workers would just grab messages from the queue as they are available.
-
-The problem you encounter is that you will not know what worker started what task.
-If you do any "management" this is a problem. For instance, if you want a task
-to have a timeout, you need to know which worker to kill if it goes over its limit.
-
-There is a way to still consolidate the call queue while no losing these other features.
-When a worker receives a task, it can submit an ACK to the finished queue telling
-the main process that it has started a task, and which task it started.
-
-This isn't ultimately robust, if there is an error between getting the message and ACK,
-but this probably isn't a reasonable concern. As of now, this looks viable.
-
-### Persistent work manager
-
-Years ago, when AWX was having trouble with output processing bottlenecks,
-we stopped using the main dispatcher process to dispatch job events to workers.
-
-Essentially, any performance-sensitive data volumes should not go through the
-pool worker management system where data is passed through IPC queues.
-Doing this causes the main process to be a bottleneck.
-
-The solution was to have workers connect to a socket on their own.
-
-Nothing is wrong with this, it's just weird.
-None of the written facilities for pool management in dispatcher code is useful.
-Because of that, event processing diverged far from the rest of the dispatcher.
-
-Long-term vision here is that:
- - a `@task` decorator may mark a task as persistent
- - additional messages types will need to be send into the finished queue for
-   - analytics tracking, like how many messages were processed
-   - whether a particular resource being monitored has been closed
+### Notable python tasking systems
 
-The idea is that this would integrate what was prototyped in:
+https://taskiq-python.github.io/guide/architecture-overview.html#context
 
-https://github.com/AlanCoding/receptor-reporter/tree/devel
+https://python-rq.org/docs/workers/
 
-That idea involved the main process more than the existing callback receiver.
-Because each job has its own socket that has to be read from, so these will come and go.
-And a worker may manage more than 1 job at the same time, asynchronously.
+https://dramatiq.io/
 
-This also requires forking from what is now `dispatcher.main`.
-We could keep the pool (and add more feature) but this requires
-an entirely different main loop.
+https://docs.celeryq.dev/