✨ Add Celery task manager to Web Server ⚠️

[giancarloromeo]

What do these changes do?

This PR adds Celery task manager functionality to the Web Server, enabling asynchronous task management and streaming capabilities. The changes refactor the existing task handling system from RabbitMQ-based async jobs to a Celery-based task manager with Redis backend.

Key Changes:

• Integrates Celery task manager with Redis backend for task storage and event streaming
• Refactors task REST API endpoints to use new Celery-based services instead of RabbitMQ async jobs

NOTE

This PR is needed for #8102 issue, allowing Streaming of events from running tasks.

Related issue/s

• related to Expose search API in storage #8102

How to test

cd services/web/server
pytest -vv --pdb tests/unit/with_dbs/01/tasks

Dev-ops

• New WEBSERVER_CELERY env var, must be enabled only on web-server and not in wb-*

[codecov]

Codecov Report

❌ Patch coverage is 74.85030% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.61%. Comparing base (c7bdff6) to head (5eb114b).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8436      +/-   ##
==========================================
- Coverage   87.64%   87.61%   -0.04%     
==========================================
  Files        1983     1990       +7     
  Lines       77292    77403     +111     
  Branches     1333     1333              
==========================================
+ Hits        67746    67820      +74     
- Misses       9147     9184      +37     
  Partials      399      399              

Flag	Coverage Δ
integrationtests	64.06% <51.12%> (-0.05%)	⬇️
unittests	86.31% <74.85%> (-0.03%)	⬇️

Components	Coverage Δ
pkg_aws_library	93.59% <ø> (ø)
pkg_celery_library	84.27% <96.77%> (+0.51%)	⬆️
pkg_dask_task_models_library	79.33% <ø> (ø)
pkg_models_library	93.07% <ø> (ø)
pkg_notifications_library	85.20% <ø> (ø)
pkg_postgres_database	87.95% <ø> (ø)
pkg_service_integration	70.17% <ø> (ø)
pkg_service_library	70.92% <0.00%> (ø)
pkg_settings_library	90.19% <ø> (ø)
pkg_simcore_sdk	84.89% <ø> (-0.18%)	⬇️
agent	93.54% <ø> (ø)
api_server	91.90% <100.00%> (ø)
autoscaling	95.71% <ø> (ø)
catalog	92.36% <ø> (ø)
clusters_keeper	99.13% <ø> (ø)
dask_sidecar	92.35% <ø> (+0.56%)	⬆️
datcore_adapter	97.94% <ø> (ø)
director	75.81% <ø> (-0.09%)	⬇️
director_v2	90.92% <ø> (-0.02%)	⬇️
dynamic_scheduler	96.71% <ø> (ø)
dynamic_sidecar	90.43% <ø> (ø)
efs_guardian	89.74% <ø> (ø)
invitations	91.41% <ø> (ø)
payments	92.71% <ø> (ø)
resource_usage_tracker	92.31% <ø> (+0.16%)	⬆️
storage	86.74% <100.00%> (ø)
webclient	∅ <ø> (∅)
webserver	87.27% <69.92%> (-0.13%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7bdff6...5eb114b. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 5eb114b.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	unit-tests	Base branch is healthy, but retries were needed. Could be early signs of flakiness 👀		1	View	View

[GitHK]

Looks good, just wondering why the stream is still present here

[pcrespov]

thx! (and sorry for the delay in the review)

[bisgaard-itis]

Thanks!

[wvangeit]

Thank you

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[giancarloromeo]

Thanks @pcrespov