🐛Storage: Listing path query grows with number of projects unbounded

[sanderegg]

What do these changes do?

fixes

that happens when the number of accessible projects grows beyond some number.

details

the cursor pagination parameter was integrating the list of projects to filter with and this growths with the number of projects.
luckily the e2e user projects never get deleted and that is why this started to happen.

the fix

• remove the list of projects from the cursor parameter,
• adds an arbitrary limit that will be easier to spot next time it happens,
• we need to join that call instead of creating the full list @matusdrobuliak66 maybe has an idea with this.
• --> Storage: access rights calls to the DB should be done using JOIN calls instead of listing everything #8544

Related issue/s

How to test

Dev-ops

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.25%. Comparing base (0b2ea9f) to head (a1ee341).
⚠️ Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (0b2ea9f) and HEAD (a1ee341). Click for more details.

HEAD has 26 uploads less than BASE

Flag	BASE (0b2ea9f)	HEAD (a1ee341)
unittests	27	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #8542       +/-   ##
===========================================
- Coverage   80.97%   67.25%   -13.73%     
===========================================
  Files        1833      838      -995     
  Lines       71465    38121    -33344     
  Branches     1344      175     -1169     
===========================================
- Hits        57872    25637    -32235     
+ Misses      13205    12427      -778     
+ Partials      388       57      -331     

Flag	Coverage Δ
integrationtests	63.91% <ø> (+2.87%)	⬆️
unittests	86.84% <100.00%> (+6.00%)	⬆️

Components	Coverage Δ
pkg_aws_library	∅ <ø> (∅)
pkg_celery_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	76.81% <ø> (+11.62%)	⬆️
agent	∅ <ø> (∅)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	77.96% <ø> (-7.38%)	⬇️
dynamic_scheduler	∅ <ø> (∅)
dynamic_sidecar	81.89% <ø> (ø)
efs_guardian	∅ <ø> (∅)
invitations	∅ <ø> (∅)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	86.84% <100.00%> (∅)
webclient	∅ <ø> (∅)
webserver	58.90% <ø> (-13.21%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b2ea9f...a1ee341. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull Request Overview

This PR addresses a bug where listing path queries grew unboundedly with the number of projects. The fix removes project_ids from the pagination cursor to prevent it from growing without bounds, since cursors are passed through REST APIs. Instead, the project_ids filter is passed directly as a function parameter with validation to ensure it doesn't exceed reasonable limits.

Key Changes:

• Removed project_ids from the pagination cursor structure to prevent unbounded growth
• Added input validation with a maximum length constraint (10,000) on filter_by_project_ids
• Added debug logging for S3 listing cursor operations

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
services/storage/src/simcore_service_storage/utils/simcore_s3_dsm_utils.py	Added debug logging for cursor operations when listing child paths in S3
services/storage/src/simcore_service_storage/modules/db/file_meta_data.py	Removed project_ids from cursor parameters, added validation with max length constraint, and updated references to use the parameter directly

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[pcrespov]

thx!

[odeimaiz]

Merci!

[matusdrobuliak66]

Thanks 👍 yes we can sometimes check the join together

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for a1ee341.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	system-tests	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		0	View	View