Skip to content

Refactoring of in-house scheduling of computational jobs to allow scaling of simcore services and reduce load on system #6666

@sanderegg

Description

@sanderegg

The scheduling of tasks from the director-v2 has the following issues:

  1. on each round all the computational jobs are checked (creating a lot of network calls to the dask-schedulers),
  2. every time a new pipeline is added 1. is done again even if it was just done,
  3. every time a pipeline is stopped 1. is done again,
  4. it is not scalable on multiple director-v2 replicas,

--> A distributed lock shall be used to protect each pipeline separately (this way multiple replicas can take care of different pipelines),
after adding/stopping a pipeline only that one should be re-scheduled, not all of them

### Tasks
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/6736

Metadata

Metadata

Assignees

Labels

a:director-v2issue related with the director-v2 service

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions