Skip to content

✨ Add a Distributed Task Queue (using Celery) (🏗️ ⚠️DEVOPS) #7214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 163 commits into from
Mar 11, 2025
Merged

✨ Add a Distributed Task Queue (using Celery) (🏗️ ⚠️DEVOPS) #7214

merged 163 commits into from
Mar 11, 2025

Conversation

@giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented Feb 12, 2025 •
edited
Loading

What do these changes do?

Introduce a Distributed Task Queue (using Celery). On the first iteration it will be used by Storage service for file compression.
The Storage now works on 2 modes: STANDARD and WORKER.

In the STANDARD mode, the service uses Celery to submit tasks and get progress/results.
In the WORKER mode, it is executed as Celery worker (with FastAPI intitialized) and it is responsible for tasks execution.

Related issue/s

How to test

Dev-ops checklist

@giancarloromeo giancarloromeo self-assigned this Feb 12, 2025
@giancarloromeo giancarloromeo added the a:storage issue related to storage service label Feb 12, 2025
@giancarloromeo giancarloromeo added this to the Singularity milestone Feb 12, 2025
@giancarloromeo giancarloromeo changed the title ✨ Add a Distribute Task Queue (using Celery) ✨ Add a Distributed Task Queue (using Celery) Feb 12, 2025
@codecov
Copy link

codecov bot commented Feb 12, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 78.21229% with 78 lines in your changes missing coverage. Please review.

Project coverage is 87.25%. Comparing base (879da9a) to head (f6232ff).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7214      +/-   ##
==========================================
+ Coverage   86.30%   87.25%   +0.95%     
==========================================
  Files        1671     1720      +49     
  Lines       64728    66572    +1844     
  Branches     1048     1157     +109     
==========================================
+ Hits        55866    58090    +2224     
+ Misses       8563     8151     -412     
- Partials      299      331      +32
Flag Coverage Δ
integrationtests 65.32% <10.00%> (-0.02%) ⬇️
unittests 86.42% <78.21%> (+1.02%) ⬆️
Components Coverage Δ
api 76.84% <ø> (ø)
pkg_aws_library 94.24% <ø> (ø)
pkg_dask_task_models_library 97.09% <ø> (ø)
pkg_models_library 91.72% <100.00%> (+0.09%) ⬆️
pkg_notifications_library 84.57% <ø> (ø)
pkg_postgres_database 88.21% <ø> (ø)
pkg_service_integration 70.03% <ø> (∅)
pkg_service_library 72.48% <ø> (ø)
pkg_settings_library 90.78% <100.00%> (+0.17%) ⬆️
pkg_simcore_sdk 85.46% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 90.68% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.12% <ø> (ø)
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.25% <ø> (ø)
datcore_adapter 98.11% <ø> (ø)
director 76.59% <ø> (ø)
director_v2 91.30% <ø> (ø)
dynamic_scheduler 97.33% <ø> (ø)
dynamic_sidecar 90.07% <ø> (ø)
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
osparc_gateway_server ∅ <ø> (∅)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.11% <ø> (-0.11%) ⬇️
storage 85.61% <75.39%> (-1.34%) ⬇️
webclient ∅ <ø> (∅)
webserver 85.80% <100.00%> (+4.64%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 879da9a...f6232ff. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@giancarloromeo giancarloromeo removed the 🤖-do-not-merge (optional) blocks Mergify from merging the PR label Mar 10, 2025
mrnicegyu11
mrnicegyu11 approved these changes Mar 10, 2025
View reviewed changes
Copy link
Member

@mrnicegyu11 mrnicegyu11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for forgetting to unblock, this is all good, thank you very much! 💪

@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 11, 2025 •
edited
Loading

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@giancarloromeo giancarloromeo enabled auto-merge (squash) March 11, 2025 10:50
@giancarloromeo giancarloromeo merged commit 60f74ff into ITISFoundation:master Mar 11, 2025
90 of 95 checks passed
@giancarloromeo giancarloromeo deleted the add-distributed-task-queue branch March 11, 2025 12:22
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-simcore that referenced this pull request Mar 26, 2025
@giancarloromeo @mrnicegyu11
✨ Add a Distributed Task Queue (using Celery) (🏗️ ⚠️DEVOPS) (ITISFoun…
3682536 
…dation#7214)
YuryHrytsuk added a commit to YuryHrytsuk/osparc-ops-environments that referenced this pull request Apr 3, 2025
@YuryHrytsuk
Add sto-worker simcore service
899965d 
Related PR: ITISFoundation/osparc-simcore#7214
@YuryHrytsuk YuryHrytsuk mentioned this pull request Apr 3, 2025
Merged
10 tasks
YuryHrytsuk added a commit to ITISFoundation/osparc-ops-environments that referenced this pull request Apr 3, 2025
@YuryHrytsuk
Add sto-worker simcore service (#1008)
611900c 
* Add sto-worker simcore service

Related PR: ITISFoundation/osparc-simcore#7214

* Tighten CPU resource constraints
This was referenced Apr 10, 2025
Closed
Closed
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Apr 15, 2025
Closed
56 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GitHK GitHK GitHK approved these changes

@mrnicegyu11 mrnicegyu11 mrnicegyu11 approved these changes

@pcrespov pcrespov pcrespov approved these changes

@sanderegg sanderegg sanderegg approved these changes

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@bisgaard-itis bisgaard-itis bisgaard-itis approved these changes

Assignees

@giancarloromeo giancarloromeo

Labels

a:storage issue related to storage service

Projects

None yet

Milestone

The Awakening

Development

Successfully merging this pull request may close these issues.

8 participants

@giancarloromeo @YuryHrytsuk @GitHK @mrnicegyu11 @pcrespov @sanderegg @matusdrobuliak66 @bisgaard-itis