[DOP-19933] - update worker, scheduler, keycloak documentation

Author: maxim-lixakov

docs/backend/auth/keycloak.rst

@@ -8,11 +8,6 @@ Description
 
 TODO:
 
-Strategies
-----------
-
-TODO:
-
 Interaction schema
 ------------------
 
@@ -79,3 +74,13 @@ Basic configuration
 .. autopydantic_model:: syncmaster.settings.auth.keycloak.KeycloakProviderSettings
 .. autopydantic_model:: syncmaster.settings.auth.jwt.JWTSettings
 
+
+Local installation and testing
+-----------------------------
+
+You can test Keycloak auth locally with docker compose:
+
+
+.. code-block:: console
+
+    $ docker compose -f docker-compose.test.yml up keycloak -d

docs/backend/install.rst

@@ -79,15 +79,11 @@ Available *extras* are:
 Run database
 ~~~~~~~~~~~~
 
-Start Postgres instance somewhere, and set up environment variables:
+Start Postgres instance somewhere, and set up environment variable:
 
 .. code-block:: bash
 
-    POSTGRES_HOST=localhost
-    POSTGRES_PORT=5432
-    POSTGRES_DB=postgres
-    POSTGRES_USER=user
-    POSTGRES_PASSWORD=password
+    SYNCMASTER__DATABASE__URL=postgresql+asyncpg://syncmaster:changeme@db:5432/syncmaster
 
 You can use virtually any database supported by `SQLAlchemy <https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls>`_,
 but the only one we really tested is Postgres.
@@ -111,33 +107,11 @@ options and commands are just the same.
 Run RabbitMQ
 ~~~~~~~~~~~~
 
-Start RabbitMQ instance somewhere, and set up environment variables:
+Start RabbitMQ instance somewhere, and set up environment variable:
 
 .. code-block:: bash
 
-    RABBITMQ_HOST=somehost
-    RABBITMQ_PORT=5672
-    RABBITMQ_USER=user
-    RABBITMQ_PASSWORD=password
-
-Run worker
-~~~~~~~~~~
-
-.. note::
-
-    Before starting the worker you need to create a queue.
-    The queue is created by sending a post request to ``/queues`` endpoint (See Swagger doc for details).
-
-to start the worker you need to run the command
-
-.. code-block:: console
-
-    $ celery -A syncmaster.worker.config.celery worker --loglevel=info --max-tasks-per-child=1 -Q queue_name
-
-.. note::
-
-    The specified celery options are given as an example, you can specify other options you need.
-
+    SYNCMASTER__BROKER__URL=amqp://guest:guest@rabbitmq:5672/
 
 Run backend
 ~~~~~~~~~~~

docs/design/entities.rst

@@ -121,7 +121,7 @@ Example:
         "started_at": "2024-01-19T16:30:07+03:00",
         "ended_at": null,
         "status": "STARTED",
-        "log_url": "https://kinaba.url/...",
+        "log_url": "https://kibana.url/...",
         "transfer_dump": {
             "transfer object JSON"
         },

docs/index.rst

@@ -17,15 +17,17 @@
     backend/install
     backend/architecture
     backend/auth/index
-    backend/openapi
     backend/configuration/index
+    backend/openapi
 
 
 .. toctree::
     :maxdepth: 2
     :caption: Worker
     :hidden:
 
+    worker/start_worker
+    worker/monitoring
     worker/configuration/index
 
 

docs/worker/monitoring.rst

@@ -0,0 +1,13 @@
+Monitoring the Celery Worker
+============================
+
+Each run in the system is linked to a log URL where the Celery worker logs are available. This log URL might point to an Elastic instance or another logging tool such as Grafana. The log URL is generated based on a template configured in the backend.
+
+The configuration parameter is:
+
+.. code-block:: bash
+
+  SYNCMASTER__SERVER__LOG_URL_TEMPLATE=https://grafana.example.com?correlation_id={{ correlation_id }}&run_id={{ run.id }}
+
+You can search for each run by either its correlation id `CORRELATION_CELERY_HEADER_ID` in http headers or the ``Run.Id``.
+

docs/worker/start_worker.rst

@@ -0,0 +1,51 @@
+Starting the Celery Worker
+==========================
+
+.. note::
+
+    Before starting the worker you need to create a queue.
+    The queue is created by sending a post request to ``/queues`` endpoint (See Swagger doc for details).
+
+
+With docker
+-----------
+
+Installation process
+~~~~~~~~~~~~~~~~~~~~
+
+Docker will download worker image of syncmaster & broker, and run them.
+Options can be set via ``.env`` file or ``environment`` section in ``docker-compose.yml``
+
+.. dropdown:: ``docker-compose.yml``
+
+    .. literalinclude:: ../../docker-compose.yml
+
+.. dropdown:: ``.env.docker``
+
+    .. literalinclude:: ../../.env.docker
+
+To start the worker container you need to run the command:
+
+.. code-block:: bash
+
+    docker compose up worker -d --wait --wait-timeout 200
+
+
+
+Without docker
+--------------
+
+To start the worker you need to run the command
+
+.. code-block:: bash
+
+    python -m celery -A syncmaster.worker.celery worker
+
+You can specify options like concurrency and queues by adding additional flags:
+
+.. code-block:: bash
+
+    celery -A -A syncmaster.worker.celery  worker --concurrency=4 --max-tasks-per-child=1 --loglevel=info
+
+
+Refer to the `Celery <https://docs.celeryq.dev/en/stable/>`_ documentation for more advanced start options.