Skip to content

Conversation

@giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented Jun 24, 2025

What do these changes do?

This PR removes timeouts on startup so that App Server initialisation can complete before starting the Celery worker itself.
Compared to the previous version, all slow initialisation procedures (e.g. DB migration) can complete... while the worker_init waits for the startup_completed_event to finalize the execution.

Clean shutdown:

image

Related issue/s

How to test

Dev-ops

@giancarloromeo giancarloromeo self-assigned this Jun 24, 2025
@giancarloromeo giancarloromeo added this to the Engage milestone Jun 24, 2025
@codecov
Copy link

codecov bot commented Jun 24, 2025

Codecov Report

Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 87.77%. Comparing base (8bc8f3c) to head (e46a266).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7962      +/-   ##
==========================================
- Coverage   87.84%   87.77%   -0.08%     
==========================================
  Files        1849     1423     -426     
  Lines       71338    59115   -12223     
  Branches     1252      624     -628     
==========================================
- Hits        62668    51887   -10781     
+ Misses       8307     7018    -1289     
+ Partials      363      210     -153     
Flag Coverage Δ
integrationtests 64.29% <ø> (+0.10%) ⬆️
unittests 86.08% <0.00%> (-0.38%) ⬇️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library 71.57% <0.00%> (+0.08%) ⬆️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.10% <ø> (-0.06%) ⬇️
agent 96.29% <ø> (ø)
api_server 92.64% <ø> (ø)
autoscaling 96.03% <ø> (ø)
catalog 92.29% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 92.35% <ø> (ø)
datcore_adapter 97.94% <ø> (ø)
director 76.73% <ø> (ø)
director_v2 91.11% <ø> (+0.01%) ⬆️
dynamic_scheduler 96.69% <ø> (ø)
dynamic_sidecar 90.09% <ø> (ø)
efs_guardian 89.65% <ø> (ø)
invitations 93.60% <ø> (ø)
payments 92.57% <ø> (ø)
resource_usage_tracker 89.00% <ø> (-0.11%) ⬇️
storage 86.31% <ø> (ø)
webclient ∅ <ø> (∅)
webserver 87.62% <ø> (+0.01%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8bc8f3c...e46a266. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@giancarloromeo giancarloromeo added bug buggy, it does not work as expected a:celery-library labels Jun 24, 2025
@giancarloromeo giancarloromeo marked this pull request as ready for review June 24, 2025 10:29
@giancarloromeo giancarloromeo requested a review from pcrespov as a code owner June 24, 2025 10:29
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻 thanks for the quick fix

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks. please check the shutdown timeout comment. thanks!

@giancarloromeo giancarloromeo changed the title 🐛 Wait for proper initialization before starting Celery worker 🐛 Use context manager for app server lifecycle in Celery workers Jun 25, 2025
@giancarloromeo
Copy link
Contributor Author

While solving the startup issue, a new one related to the shutdown raised (see: #7968). The redesign solves both.

@giancarloromeo giancarloromeo requested a review from pcrespov June 25, 2025 05:56
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@sonarqubecloud
Copy link

Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thanks a lot for the quick fix

@giancarloromeo giancarloromeo merged commit 615c826 into ITISFoundation:master Jun 25, 2025
95 of 97 checks passed
@giancarloromeo giancarloromeo deleted the is7957/fix-storage-worker-startup branch June 25, 2025 07:21
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a:celery-library bug buggy, it does not work as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storage worker shuts down unexpectedly sto-worker service is failing fast when database version is not (yet) migrated

5 participants