Skip to content

✨Storage: provide entrypoint to compute path size (⚠️ devops) #7286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 42 commits into from
Mar 7, 2025
Merged

✨Storage: provide entrypoint to compute path size (⚠️ devops) #7286

merged 42 commits into from
Mar 7, 2025

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Feb 26, 2025 •
edited
Loading

What do these changes do?

This PR introduces a new API entrypoint in storage to compute an arbitrary path size. This typically will be used by the frontend to compute the size of a study, or the size of a node, or any subfolders within a node.
This works for both simcore and datcore (also this is not optimized unless the base dataset size is computed).

This PR brings the new functionality but is NOT YET available to the frontend.

Next steps

Changes

  • additional entrypoint in datcore-adapter to retrieve a single dataset and its size if available
  • new entrypoint in storage API POST /v0/locations/{location_id}/paths/{path}:size ReDoc
  • refactoring of storage testing to include DATCORE
  • removed BF_API_KEY and BF_API_SECRET, maybe the oldest unused ENVs ever

Related issue/s

How to test

Dev-ops checklist

(NON-REALLY)BREAKING CHANGES:

@sanderegg sanderegg added a:storage issue related to storage service a:webserver webserver's codebase. Assigning the area is particularly useful for bugs labels Feb 26, 2025
@sanderegg sanderegg added this to the The Awakening milestone Feb 26, 2025
@sanderegg sanderegg self-assigned this Feb 26, 2025
@codecov
Copy link

codecov bot commented Feb 26, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 76.85950% with 28 lines in your changes missing coverage. Please review.

Project coverage is 88.13%. Comparing base (5075fe0) to head (fce2443).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7286      +/-   ##
==========================================
+ Coverage   87.32%   88.13%   +0.81%     
==========================================
  Files        1693     1669      -24     
  Lines       65780    64663    -1117     
  Branches     1121     1121              
==========================================
- Hits        57441    56993     -448     
+ Misses       8019     7350     -669     
  Partials      320      320
Flag Coverage Δ
integrationtests 65.40% <ø> (-0.10%) ⬇️
unittests 86.59% <76.85%> (+0.13%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library 94.24% <ø> (ø)
pkg_dask_task_models_library 97.09% <ø> (ø)
pkg_models_library 91.63% <100.00%> (+0.01%) ⬆️
pkg_notifications_library 84.57% <ø> (ø)
pkg_postgres_database 88.28% <ø> (ø)
pkg_service_integration 70.03% <ø> (ø)
pkg_service_library 72.48% <ø> (ø)
pkg_settings_library 90.61% <ø> (ø)
pkg_simcore_sdk 85.46% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 90.68% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.10% <ø> (ø)
clusters_keeper 99.24% <ø> (ø)
dask_sidecar 91.25% <ø> (ø)
datcore_adapter 98.11% <100.00%> (+0.04%) ⬆️
director 76.59% <ø> (ø)
director_v2 91.30% <ø> (-0.13%) ⬇️
dynamic_scheduler 97.33% <ø> (ø)
dynamic_sidecar 90.09% <ø> (+0.01%) ⬆️
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
osparc_gateway_server ∅ <ø> (∅)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.11% <ø> (+0.21%) ⬆️
storage 86.81% <71.42%> (+2.56%) ⬆️
webclient ∅ <ø> (∅)
webserver 88.17% <ø> (+2.36%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5075fe0...fce2443. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch 7 times, most recently from 7856550 to 3356fab Compare March 4, 2025 20:58
@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch 2 times, most recently from 5fd55d1 to 47619af Compare March 6, 2025 09:37
@sanderegg sanderegg marked this pull request as ready for review March 6, 2025 10:16
pcrespov
pcrespov approved these changes Mar 6, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@sanderegg sanderegg changed the title ✨Storage: provide entrypoint to compute total size ✨Storage: provide entrypoint to compute path size Mar 6, 2025
@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch from f051c79 to 3f55a2a Compare March 6, 2025 13:47
@sanderegg sanderegg mentioned this pull request Mar 6, 2025
Merged
1 task
@sanderegg sanderegg changed the title ✨Storage: provide entrypoint to compute path size ✨Storage: provide entrypoint to compute path size (⚠️ devops) Mar 6, 2025
@sanderegg sanderegg requested review from GitHK and bisgaard-itis March 6, 2025 16:03
@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch from 3f55a2a to 4f2f270 Compare March 6, 2025 16:06
GitHK
GitHK approved these changes Mar 6, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch from 4f2f270 to e3215d6 Compare March 6, 2025 16:30
@sanderegg sanderegg force-pushed the storage/compute-path-total-size branch from e3215d6 to fce2443 Compare March 7, 2025 09:18
@sonarqubecloud
Copy link

sonarqubecloud bot commented Mar 7, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@sanderegg sanderegg merged commit 2415e78 into ITISFoundation:master Mar 7, 2025
192 of 199 checks passed
@sanderegg sanderegg deleted the storage/compute-path-total-size branch March 7, 2025 12:48
mrnicegyu11 pushed a commit to mrnicegyu11/osparc-simcore that referenced this pull request Mar 26, 2025
@sanderegg @mrnicegyu11
✨Storage: provide entrypoint to compute path size (⚠️ devops) (ITISFo…
7e009a0 
…undation#7286)
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Apr 15, 2025
Closed
56 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcrespov pcrespov pcrespov approved these changes

@GitHK GitHK GitHK approved these changes

@mrnicegyu11 mrnicegyu11 mrnicegyu11 approved these changes

@YuryHrytsuk YuryHrytsuk YuryHrytsuk approved these changes

@bisgaard-itis bisgaard-itis bisgaard-itis approved these changes

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66 matusdrobuliak66 is a code owner

@giancarloromeo giancarloromeo Awaiting requested review from giancarloromeo giancarloromeo is a code owner

Assignees

@sanderegg sanderegg

Labels

a:storage issue related to storage service a:webserver webserver's codebase. Assigning the area is particularly useful for bugs

Projects

None yet

Milestone

The Awakening

Development

Successfully merging this pull request may close these issues.

Study total size

6 participants

@sanderegg @GitHK @mrnicegyu11 @pcrespov @YuryHrytsuk @bisgaard-itis