[DOP-25282] Sync documentation structure with Data.Rentgen

Author: dolfinus

docs/index.rst

@@ -11,33 +11,15 @@
 
 .. toctree::
     :maxdepth: 2
-    :caption: Server
+    :caption: Reference
     :hidden:
 
-    server/install
-    server/architecture
-    server/auth/index
-    server/configuration/index
-    server/openapi
-
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Worker
-    :hidden:
-
-    worker/install
-    worker/log_url
-    worker/configuration/index
-
-
-.. toctree::
-    :maxdepth: 2
-    :caption: Scheduler
-    :hidden:
-
-    scheduler/install
-    scheduler/configuration/index
+    reference/architecture
+    reference/database/index
+    reference/broker/index
+    reference/server/index
+    reference/worker/index
+    reference/scheduler/index
 
 .. toctree::
     :maxdepth: 2

docs/reference/architecture.rst

@@ -0,0 +1,21 @@
+.. _architecture:
+
+Architecture
+============
+
+Components
+----------
+
+SyncMaster contains the following components:
+
+* :ref:`frontend`, main user interface.
+* :ref:`server`, providing REST API for fetching and manipulating entities.
+* :ref:`worker`, performing actual transfer work (ETL processes).
+* :ref:`scheduler`, scheduling transfers to be executed in future.
+* :ref:`database` for storing internal data.
+* :ref:`message-broker` for communications between Server/Scheduler and Worker.
+
+Architecture diagram
+--------------------
+
+.. image:: ../_static/architecture.png

docs/reference/broker/index.rst

@@ -0,0 +1,48 @@
+.. _message-broker:
+
+Message Broker
+==============
+
+Message broker is componen used by :ref:`server`/:ref:`scheduler` to communicate with :ref:`worker`.
+
+SyncMaster can work virtually with any broker supported by `Celery <https://docs.celeryq.dev>`_.
+But the only broker we tested is `RabbitMQ <https://www.rabbitmq.com/>`_.
+
+Requirements
+------------
+
+* RabbitMQ 4.x. It is recommended to use latest RabbitMQ version.
+
+Setup
+~~~~~
+
+With Docker
+^^^^^^^^^^^
+
+* Install `Docker <https://docs.docker.com/engine/install/>`_
+* Install `docker-compose <https://github.com/docker/compose/releases/>`_
+* Run the following command:
+
+  .. code:: console
+
+    $ docker compose --profile broker up -d --wait
+
+  ``docker-compose`` will download RabbitMQ image, create container and volume, and then start container.
+  Image entrypoint will create database if volume is empty.
+
+  Options can be set via ``.env`` file or ``environment`` section in ``docker-compose.yml``
+
+  .. dropdown:: ``docker-compose.yml``
+
+    .. literalinclude:: ../../../docker-compose.yml
+        :emphasize-lines: 33-45,144
+
+  .. dropdown:: ``.env.docker``
+
+    .. literalinclude:: ../../../.env.docker
+        :emphasize-lines: 15-16
+
+Without Docker
+^^^^^^^^^^^^^^
+
+Please follow `RabbitMQ installation instruction <https://www.rabbitmq.com/docs/download>`_.

docs/reference/database/configuration.rst

@@ -0,0 +1,6 @@
+.. _configuration-database:
+
+Database settings
+=================
+
+.. autopydantic_model:: syncmaster.settings.DatabaseSettings

docs/reference/database/credentials_encryption.rst

@@ -0,0 +1,6 @@
+.. _configuration-credentials-encryption:
+
+Credentials encryption
+======================
+
+.. autopydantic_model:: syncmaster.settings.credentials.CredentialsEncryptionSettings

docs/reference/database/index.rst

@@ -0,0 +1,109 @@
+.. _database:
+
+Relation Database
+=================
+
+SyncMaster requires relational database for storing internal data.
+
+Currently, SyncMaster supports only `PostgreSQL <https://www.postgresql.org/>`_ as storage for lineage entities and relations.
+
+Migrations
+------------
+
+After a database is started, it is required to run migration script.
+For empty database, it creates all the required tables and indexes.
+For non-empty database, it will perform database structure upgrade, using `Alembic <https://alembic.sqlalchemy.org/>`_.
+
+.. warning::
+
+    Other containers (server, scheduler, worker) should be stopped while running migrations, to prevent interference.
+
+Requirements
+------------
+
+* PostgreSQL 12 or higher. It is recommended to use latest Postgres version.
+
+Install & run
+-------------
+
+With Docker
+~~~~~~~~~~~
+
+* Install `Docker <https://docs.docker.com/engine/install/>`_
+* Install `docker-compose <https://github.com/docker/compose/releases/>`_
+
+* Run the following command:
+
+  .. code:: console
+
+    $ docker compose up -d db db-migrations
+
+  ``docker-compose`` will download PostgreSQL image, create container and volume, and then start container.
+  Image entrypoint will create database if volume is empty.
+
+  After that, one-off container with migrations script will run.
+
+  Options can be set via ``.env`` file or ``environment`` section in ``docker-compose.yml``
+
+  .. dropdown:: ``docker-compose.yml``
+
+      .. literalinclude:: ../../../docker-compose.yml
+          :emphasize-lines: 1-31,142
+
+  .. dropdown:: ``.env.docker``
+
+      .. literalinclude:: ../../../.env.docker
+          :emphasize-lines: 8-9
+
+Without Docker
+~~~~~~~~~~~~~~
+
+* For installing PostgreSQL, please follow `installation instruction <https://www.postgresql.org/download/>`_.
+* Install Python 3.11 or above
+* Create virtual environment
+
+  .. code-block:: console
+
+      $ python -m venv /some/.venv
+      $ source /some/.venv/activate
+
+* Install ``syncmaster`` package with following *extra* dependencies:
+
+  .. code-block:: console
+
+      $ pip install syncmaster[postgres]
+
+* Configure :ref:`Database connection <configuration-database>` using environment variables, e.g. by creating ``.env`` file:
+
+  .. code-block:: console
+    :caption: /some/.env
+
+    $ export SYNCMASTER__DATABASE__URL=postgresql+asyncpg://syncmaster:changeme@db:5432/syncmaster
+
+  And then read values from this file:
+
+  .. code:: console
+
+    $ source /some/.env
+
+* Run migrations:
+
+  .. code:: console
+
+    $ python -m syncmaster.db.migrations upgrade head
+
+  This is a thin wrapper around `alembic cli <https://alembic.sqlalchemy.org/en/latest/tutorial.html#running-our-first-migration>`_,
+  options and commands are just the same.
+
+  .. note::
+
+      This command should be executed after each upgrade to new Data.Rentgen version.
+
+See also
+--------
+
+.. toctree::
+    :maxdepth: 1
+
+    configuration
+    credentials_encryption