🎨 No more long running http requests while stopping services #8531

GitHK · 2025-10-20T10:32:48Z

What do these changes do?

Changes how the stop procedure works. It no longer involves long running http connections.
Stop returns immediately and the status of the service is monitor until it becomes idle, marking the removal of the service.
This approach works for both legacy and new style dynamic services.

The dynamic-scheduler now creates a fire_and_forget task to deal with stopping legacy services, since it is still required to wait to up to 1 hour for these to finish stopping.

Related issue/s

How to test

Dev-ops

codecov · 2025-10-20T10:34:39Z

Codecov Report

❌ Patch coverage is 68.62745% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.32%. Comparing base (a7cae35) to head (c86c138).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8531      +/-   ##
==========================================
- Coverage   87.52%   87.32%   -0.21%     
==========================================
  Files        2009     1578     -431     
  Lines       78515    65660   -12855     
  Branches     1344      682     -662     
==========================================
- Hits        68721    57338   -11383     
+ Misses       9392     8082    -1310     
+ Partials      402      240     -162

Flag	Coverage Δ
integrationtests	`63.90% <30.76%> (-0.05%)`	⬇️
unittests	`85.78% <68.62%> (-0.46%)`	⬇️

Components	Coverage Δ
pkg_aws_library	`∅ <ø> (∅)`
pkg_celery_library	`∅ <ø> (∅)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`∅ <ø> (∅)`
pkg_notifications_library	`∅ <ø> (∅)`
pkg_postgres_database	`∅ <ø> (∅)`
pkg_service_integration	`∅ <ø> (∅)`
pkg_service_library	`70.94% <ø> (ø)`
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`84.95% <ø> (ø)`
agent	`93.10% <ø> (ø)`
api_server	`91.62% <ø> (ø)`
autoscaling	`95.00% <ø> (ø)`
catalog	`92.06% <ø> (ø)`
clusters_keeper	`99.14% <ø> (ø)`
dask_sidecar	`92.38% <ø> (ø)`
datcore_adapter	`97.95% <ø> (ø)`
director	`75.72% <ø> (ø)`
director_v2	`90.88% <ø> (-0.02%)`	⬇️
dynamic_scheduler	`96.54% <81.57%> (-0.28%)`	⬇️
dynamic_sidecar	`90.44% <ø> (ø)`
efs_guardian	`89.83% <ø> (ø)`
invitations	`90.90% <ø> (ø)`
payments	`92.80% <ø> (ø)`
resource_usage_tracker	`91.79% <ø> (-0.43%)`	⬇️
storage	`86.97% <ø> (+0.08%)`	⬆️
webclient	`∅ <ø> (∅)`
webserver	`87.06% <30.76%> (-0.02%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7cae35...c86c138. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mergify · 2025-10-20T11:10:39Z

🧪 CI Insights

Here's what we observed from your CI run for c86c138.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on `master`	Retries	🔍 CI Insights	📄 Logs
`CI`	`system-tests`	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		`0`	View	View

…shutdown

Copilot

Pull Request Overview

This PR refactors the dynamic service stop mechanism to use an asynchronous polling approach instead of long-running HTTP connections. The stop operation now returns immediately and monitors the service status until it reaches an idle state. For legacy services that may take up to 1 hour to stop, the dynamic-scheduler creates fire-and-forget tasks to handle the stopping process without blocking.

Key changes:

Webserver now polls service status after initiating stop instead of waiting on a long HTTP connection
Dynamic-scheduler uses fire-and-forget tasks for legacy service stops that may take extended time
Removed timeout parameter from RPC interface as stops are now non-blocking

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/web/server/src/simcore_service_webserver/dynamic_scheduler/api.py	Implements polling mechanism to wait for service to become idle after stop request
services/dynamic-scheduler/tests/unit/api_rpc/test_api_rpc__services.py	Removes timeout_s parameter from stop_dynamic_service test calls
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/fire_and_froget.py	Adds new FireAndForgetCollection to manage background tasks
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/director_v2/_public_client.py	Adds cleanup of public client from app state
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/common_interface.py	Distinguishes legacy vs new-style services and uses fire-and-forget for legacy stops
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/catalog/_public_client.py	Renames get_services_labels to get_service_labels
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/core/events.py	Registers fire_and_forget_lifespan
services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/api/frontend/routes_external_scheduler/_service.py	Removes unused timeout_s parameter from service_stop
services/director-v2/src/simcore_service_director_v2/core/dynamic_services_settings/scheduler.py	Updates comment to clarify timeout applies to LEGACY services
services/director-v2/src/simcore_service_director_v2/api/routes/dynamic_services.py	Removes polling logic from stop endpoint as it's now handled by caller
packages/service-library/src/servicelib/rabbitmq/rpc_interfaces/dynamic_scheduler/services.py	Removes timeout_s parameter from stop_dynamic_service RPC interface

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/fire_and_froget.py

...ler/src/simcore_service_dynamic_scheduler/api/frontend/routes_external_scheduler/_service.py

...ices/director-v2/src/simcore_service_director_v2/core/dynamic_services_settings/scheduler.py

sonarqubecloud · 2025-10-21T09:02:38Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Copilot

Pull Request Overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/fire_and_froget.py

Copilot · 2025-10-21T09:06:40Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/common_interface.py

+from .catalog._public_client import CatalogPublicClient
 from .director_v2 import DirectorV2Client
-from .service_tracker import set_request_as_running, set_request_as_stopped
+from .fire_and_froget import FireAndForgetCollection


Import path contains typo: 'froget' should be 'forget'.

Suggested change

from .fire_and_froget import FireAndForgetCollection

from .fire_and_forget import FireAndForgetCollection

this is still wrong

Copilot · 2025-10-21T09:06:40Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/core/events.py

 from ..services.deferred_manager import deferred_manager_lifespan
 from ..services.director_v0 import director_v0_lifespan
 from ..services.director_v2 import director_v2_lifespan
+from ..services.fire_and_froget import fire_and_forget_lifespan


Import path contains typo: 'froget' should be 'forget'.

Suggested change

from ..services.fire_and_froget import fire_and_forget_lifespan

from ..services.fire_and_forget import fire_and_forget_lifespan

this is still mispelled (the first part)

pcrespov

thx. Left some comments.

I recommend you to cleanup naming typos, scheck suggestions and I would also make sure has a good test coverage

pcrespov · 2025-10-21T13:09:58Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/core/events.py

 from ..services.deferred_manager import deferred_manager_lifespan
 from ..services.director_v0 import director_v0_lifespan
 from ..services.director_v2 import director_v2_lifespan
+from ..services.fire_and_froget import fire_and_forget_lifespan


this is still mispelled (the first part)

pcrespov · 2025-10-21T13:16:44Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/common_interface.py

+from .catalog._public_client import CatalogPublicClient
 from .director_v2 import DirectorV2Client
-from .service_tracker import set_request_as_running, set_request_as_stopped
+from .fire_and_froget import FireAndForgetCollection


this is still wrong

pcrespov · 2025-10-21T13:24:35Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/fire_and_froget.py

name of this file "froget"

pcrespov · 2025-10-21T13:30:08Z

services/web/server/src/simcore_service_webserver/dynamic_scheduler/api.py



+def _wait_for_idle_retry_error(node_id: NodeID, retry_state: RetryCallState):
+    _logger.info(


TIP: you are hooking logs to before_sleep and retry_error_callback

Note that tenacity has paramters for exaclyt that called: before_log, after_log and before_sleep_log

https://tenacity.readthedocs.io/en/latest/#before-and-after-retry-and-logging

sanderegg

thanks

sanderegg · 2025-10-21T13:36:47Z

services/dynamic-scheduler/src/simcore_service_dynamic_scheduler/services/common_interface.py

+    tracked_service = await get_tracked_service(app, dynamic_service_stop.node_id)
+
+    if tracked_service and tracked_service.dynamic_service_start:
+        service_labels = await CatalogPublicClient.get_from_app_state(


if the service is there, why do you need to ask the catalog about these labels?

sanderegg · 2025-10-21T13:38:59Z

services/web/server/src/simcore_service_webserver/dynamic_scheduler/api.py

                settings.DYNAMIC_SCHEDULER_STOP_SERVICE_TIMEOUT.total_seconds()
            ),
-        )
+            before_sleep=partial(


hmm you are maybe looking for before_sleep_log?

changed how stop works

5b114e6

GitHK self-assigned this Oct 20, 2025

GitHK added this to the Imparable milestone Oct 20, 2025

fixed borken tests

3783629

GitHK added the t:maintenance Some planned maintenance work label Oct 20, 2025

Andrei Neagu added 9 commits October 20, 2025 15:58

restore timeout for stopping services

0a59492

updated for legacy

dd98ef3

refactor

57d817a

refactor stop services

a8cf58b

refactor log messages

4f9294e

unfied the itnerface a bit more

82c67e0

Merge remote-tracking branch 'upstream/master' into pr-osparc-rework-…

774e1cd

…shutdown

dynamic-scheduler runs long_running legacy services stop call request

d509a12

fixed shutdown issues with legacy

4964f20

GitHK added the bug buggy, it does not work as expected label Oct 21, 2025

GitHK requested a review from Copilot October 21, 2025 08:54

Copilot AI reviewed Oct 21, 2025

View reviewed changes

Andrei Neagu added 2 commits October 21, 2025 10:58

typos

3004f6d

renamed

c86c138

GitHK requested a review from Copilot October 21, 2025 09:05

Copilot AI reviewed Oct 21, 2025

View reviewed changes

GitHK changed the title ~~🎨 Changed how service stop works~~ 🎨 No more long running http requests while stopping services Oct 21, 2025

GitHK marked this pull request as ready for review October 21, 2025 09:36

GitHK requested review from matusdrobuliak66, pcrespov and sanderegg as code owners October 21, 2025 09:36

GitHK requested a review from bisgaard-itis October 21, 2025 09:37

GitHK requested a review from giancarloromeo October 21, 2025 09:37

pcrespov approved these changes Oct 21, 2025

View reviewed changes

sanderegg approved these changes Oct 21, 2025

View reviewed changes

	from .fire_and_froget import FireAndForgetCollection
	from .fire_and_forget import FireAndForgetCollection

	from ..services.fire_and_froget import fire_and_forget_lifespan
	from ..services.fire_and_forget import fire_and_forget_lifespan



		def _wait_for_idle_retry_error(node_id: NodeID, retry_state: RetryCallState):
		_logger.info(

🎨 No more long running http requests while stopping services #8531

Are you sure you want to change the base?

🎨 No more long running http requests while stopping services #8531

Uh oh!

Conversation

GitHK commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue/s

How to test

Dev-ops

Uh oh!

codecov bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mergify bot commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 CI Insights

✅ Passed Jobs With Interesting Signals

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Oct 21, 2025

Quality Gate passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcrespov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GitHK commented Oct 20, 2025 •

edited

Loading

codecov bot commented Oct 20, 2025 •

edited

Loading

mergify bot commented Oct 20, 2025 •

edited

Loading