✨ dynamic-scheduler can chain operations

[GitHK]

What do these changes do?

In order to support meaningful scheduling workflows the ability to start operations one after another is required.

Operations can be:

• register to start for: on_create_completed (after the CREATE phase completes with success) and on_undo_completed (after the UNDO phase completes with success)
• the followup operation can be registered either when running start_operation (which now accepts extra optional parameters) or after it is already started via register_to_start_after_on_created_completed or register_to_start_after_on_undo_completed

Enhancements:

• Operation is now an object, which allows to declare if it can be cancelled or not, since it does not make sense to cancel certain ones.
• Fixed 3 flaky tests inside services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__core.py by checking for the presence of a field instead of randomly waiting

Related issue/s

• Stop button to get services out of the failed state. (Can probably be linked to #1045) osparc-issues#1167
• Make director-v2 scalable/restartable #4524
• dynamic scheduler: extract scheduling logic and move it away from director-v2 #5632

How to test

Dev-ops

[codecov]

Codecov Report

❌ Patch coverage is 98.86364% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.08%. Comparing base (ef6f3f1) to head (a1338ac).

❗ There is a different number of reports uploaded between BASE (ef6f3f1) and HEAD (a1338ac). Click for more details.

HEAD has 31 uploads less than BASE

Flag	BASE (ef6f3f1)	HEAD (a1338ac)
unittests	32	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #8446       +/-   ##
===========================================
- Coverage   87.61%   68.08%   -19.53%     
===========================================
  Files        1990      850     -1140     
  Lines       77403    37756    -39647     
  Branches     1333      175     -1158     
===========================================
- Hits        67814    25706    -42108     
- Misses       9190    11993     +2803     
+ Partials      399       57      -342     

Flag	Coverage Δ
integrationtests	64.06% <ø> (-0.04%)	⬇️
unittests	96.84% <98.86%> (+10.53%)	⬆️

Components	Coverage Δ
pkg_aws_library	∅ <ø> (∅)
pkg_celery_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	76.69% <ø> (-8.26%)	⬇️
agent	∅ <ø> (∅)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	78.04% <ø> (-12.87%)	⬇️
dynamic_scheduler	96.84% <98.86%> (+0.13%)	⬆️
dynamic_sidecar	81.87% <ø> (-8.56%)	⬇️
efs_guardian	∅ <ø> (∅)
invitations	∅ <ø> (∅)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	∅ <ø> (∅)
webclient	∅ <ø> (∅)
webserver	59.05% <ø> (-28.20%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef6f3f1...a1338ac. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for a1338ac.

🟢 All jobs passed!

But CI Insights is watching 👀

[Copilot]

Pull Request Overview

This PR continues the migration of the dynamic scheduler module (part 6), focusing on enhancing the generic scheduler with event-driven capabilities and operation chaining functionality.

Key changes include:

• Added event-driven architecture for operation completion notifications
• Introduced operation chaining functionality allowing operations to start after others complete
• Enhanced the Operation class to support cancellation control

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
services/dynamic-scheduler/tests/unit/services/generic_scheduler/utils.py	Added utility functions for Redis store key validation and helper imports
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test_generic_scheduler.py	Enhanced tests with operation chaining scenarios and event handling validation
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__store.py	Updated store method names and added event proxy tests
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__operation.py	Updated Operation instantiation from list to class constructor
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__event_scheduler.py	Added comprehensive event scheduling tests for create/undo completion events
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__event_after.py	New test file for after-event management functionality
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__deferred_runner.py	Updated Operation instantiation and disabled redis-commander
services/dynamic-scheduler/tests/unit/services/generic_scheduler/test__core.py	Extensive updates with new Operation syntax, event handling tests, and store validation
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_store.py	Added event proxy classes and renamed store methods for consistency
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_operation.py	Converted Operation from TypeAlias to class with cancellation support
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_models.py	Added EventType enum and OperationToStart dataclass
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_lifespan.py	Integrated AfterEventManager into application lifecycle
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_event_scheduler.py	Enhanced with create/undo completion event handling
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_event_after_registration.py	New module for event registration functions
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_event_after.py	New module implementing after-event management
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_event.py	Enhanced event functions for create/undo completion events
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_errors.py	Added OperationNotCancellableError
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/_core.py	Major updates with event emission, operation chaining, and cancellation control
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/generic_scheduler/init.py	Updated exports for new event registration functions

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[pcrespov]

HEre a firest round of suggestions. I have some questions but i better do it offline.

[pcrespov]

thx

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud