🎨Storage: HA (🚨🚨🚨 test with multiple replicas)

[sanderegg]

What do these changes do?

This PR shall unlock having multiple replicas of storage micro-service by:

• moving copying of files to the storage worker via Celery
• copying of files is now triggered via RPC instead of REST (the REST API POST /simcore-s3/folders is gone)
• moving completion of multipart upload from in-process task to Celery task, e.g. offloaded to the storage worker (here the REST API is kept for backwards compatibility + it is already a long running task)
• unit tests now run the storage manager + celery (with in-memory transfer) + storage worker in the same process to allow testing (there is still leeway to improve this) see test_rpc_handlers_paths
• --> now testing storage also need the storage worker to be up

Bonus:

• fixes shaky storage worker: wait for the fastapi initialization to complete

Related issue/s

• fixes Storage: Handle multiple replicas of storage #7196

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 82.91139% with 27 lines in your changes missing coverage. Please review.

Project coverage is 88.03%. Comparing base (5bbcc74) to head (e39a883).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7375      +/-   ##
==========================================
+ Coverage   87.43%   88.03%   +0.60%     
==========================================
  Files        1719     1337     -382     
  Lines       66692    55309   -11383     
  Branches     1132      586     -546     
==========================================
- Hits        58310    48692    -9618     
+ Misses       8061     6431    -1630     
+ Partials      321      186     -135     

Flag	Coverage Δ
integrationtests	65.22% <43.75%> (+0.06%)	⬆️
unittests	86.21% <82.91%> (-0.40%)	⬇️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	72.64% <0.00%> (-0.23%)	⬇️
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.46% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	90.76% <ø> (ø)
autoscaling	96.08% <ø> (ø)
catalog	91.82% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.25% <ø> (ø)
datcore_adapter	98.11% <ø> (ø)
director	76.70% <ø> (+0.09%)	⬆️
director_v2	91.28% <ø> (-0.02%)	⬇️
dynamic_scheduler	97.33% <ø> (ø)
dynamic_sidecar	90.12% <ø> (+0.01%)	⬆️
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <ø> (ø)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.12% <ø> (-0.17%)	⬇️
storage	86.61% <97.56%> (+0.17%)	⬆️
webclient	∅ <ø> (∅)
webserver	88.21% <68.75%> (+2.37%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bbcc74...e39a883. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[YuryHrytsuk]

Related to #5621

[YuryHrytsuk]

Thanks 🙏 Very much appreciated 🎉

Please, once this is deployed, merge changes in OPS (including related config changes). Or just let me know, I will do it 🚀

ITISFoundation/osparc-ops-environments#1000

[GitHK]

Looking good. Have 1 question

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud