What do these changes do?

This PR heavily refactors the director-v2 internal computational scheduler by using tools to allow multiple director-v2 replicas to be able to share the load of scheduling computational pipelines.

As reminder the computational scheduler in the director-v2 is responsible for:

1. analyzing a project's computational pipeline and to find which nodes need to be (re-)run when the user starts a computation,
2. scheduling computational jobs according to the computational pipeline (what can run and what shall wait, retrieving results/errors and transmitting progress) by communicating with the shared Dask cluster or with private Dask clusters

Until this PR, replicating the director-v2 would also duplicate network calls and end in wasted resources.

This PR aims to make the replication of the director-v2 more efficient by:

• having a Manager that periodically schedules (5s) the pipelines, relying on the Database information directly, exclusively (e.g. only 1 director-v2 replica can do it at a time - using distributed locks),
• these tasks are published in a RabbitMQ shared queue named simcore.services.director-v2.scheduling,
• each director-v2 replica contains COMPUTATIONAL_BACKEND_SCHEDULING_CONCURRENCY workers that apply the scheduling and communicate with the deeper backend (e.g. Dask), this allows to share the load between the different workers, so in effect every new replica of the director-v2 shall relieve the other ones,
• COMPUTATIONAL_BACKEND_SCHEDULING_CONCURRENCY is currently hard-coded to 50. Some testing will be necessary to see whether that is too low or too high. That is why this is not at the moment an ENV variable. It will be converted if necessary.
• 🗃️: comp_runs table is upgraded to contain new nullable scheduled and processed columns, this is used to keep track of when a pipeline was scheduled by the manager, and when the worker has processed it.
• 🗃️: comp_runs table is upgraded to contain only timezone-enabled timestamps
• 🗃️: comp_tasks table is upgraded to contain only timezone-enabled timestamps

Schematic

---
config:
  theme: mc
  layout: dagre
  look: handDrawn
---
flowchart LR
 subgraph s1["Director-v2.1"]
        n1["Manager:<br>5s: Schedule all pipelines"]
        n3["Worker1"]
        n4["Worker2"]
        n5["WorkerN"]
        n6["schedule pipeline1"]
        n7["schedule pipeline2"]
        n8["schedule pipeline3"]
  end
 subgraph s2["Cluster-UserX"]
        n9["Dask-Scheduler"]
        n11["Dask-Worker(s)"]
  end
 subgraph s3["Cluster-UserY"]
        n14["Dask-Scheduler"]
        n15["Dask-Worker(s)"]
  end
 subgraph s4["Cluster-UserZ"]
        n19["Dask-Scheduler"]
        n20["Dask-Worker(s)"]
  end
 subgraph s5["Director-v2.2"]
        n21["Manager:<br>5s: Schedule all pipelines"]
        n22["Worker1"]
        n23["Worker2"]
        n24["WorkerN"]
        n25["schedule pipeline4"]
        n26["schedule pipeline5"]
        n27["schedule pipeline6"]
  end
 subgraph s6["Cluster-UserA"]
        n90["Dask-Scheduler"]
        n91["Dask-Worker(s)"]
  end
 subgraph s7["Cluster-UserB"]
        n92["Dask-Scheduler"]
        n93["Dask-Worker(s)"]
  end
 subgraph s8["Cluster-UserC"]
        n94["Dask-Scheduler"]
        n95["Dask-Worker(s)"]
  end
    n1 ==> n2["RabbitMQ"]
    n2 --> n3 & n4 & n5 & n22 & n23 & n24
    n3 --> n6
    n4 --> n7
    n5 --> n8
    n6 --> n9
    n7 --> n14
    n8 --> n19
    n22 --> n25
    n23 --> n26
    n24 --> n27
    n21 ==> n2
    n25 --> n90
    n26 --> n92
    n27 --> n94
    n21@{ shape: rect}
    n22@{ shape: rect}
    n23@{ shape: rect}
    n24@{ shape: rect}
    n25@{ shape: rect}
    n26@{ shape: rect}
    n27@{ shape: rect}
    n2@{ shape: cyl}
    style n1 stroke:#D50000
    style n6 stroke:#D50000
    style n7 stroke:#D50000
    style n8 stroke:#D50000
    style n21 stroke:#D50000,stroke-width:1px,stroke-dasharray: 1
    style n25 stroke:#D50000
    style n26 stroke:#D50000
    style n27 stroke:#D50000

Legend:

• in $${\color{red}red}$$ are blocks that run exclusively via Redis locks

Related issue/s

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)