🐛 Fix performance issue when listing files filtering on projects

[bisgaard-itis]

What do these changes do?

• Fix a clause in storage when filtering on wither user_id or project_ids.
• The bug is that when UserOrProjectFilter.user_id == None then the filtering on project_ids is not taken into account. This PR fixes that.
• Note that this fix only affects the case user_id==None which can only ever happen here: https://github.com/ITISFoundation/osparc-simcore/blob/master/services/storage/src/simcore_service_storage/simcore_s3_dsm.py#L363. And as can be seen from the logic there this will happen only when project_id is not None. In that case we need to to use the SQL index on the project_id column in the file_meta_data table to do the search.

Related issue/s

• This issue was detected as part of the investigation into [Bug]: on osparc.io it seems a user can only run 1 function map at a time #8438 (comment). It is still not entirely clear where the meany returned rows in that issue comes from. But at least this PR should fix one of the performance bottlenecks causing the slow SQL queries.

How to test

Dev-ops

[wvangeit]

Thanks

[matusdrobuliak66]

Thank you 💯

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.47%. Comparing base (699808f) to head (5d27ec8).
⚠️ Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (699808f) and HEAD (5d27ec8). Click for more details.

HEAD has 31 uploads less than BASE

Flag	BASE (699808f)	HEAD (5d27ec8)
unittests	32	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #8443       +/-   ##
===========================================
- Coverage   87.91%   67.47%   -20.45%     
===========================================
  Files        1976      822     -1154     
  Lines       77221    37609    -39612     
  Branches     1342      175     -1167     
===========================================
- Hits        67889    25375    -42514     
- Misses       8928    12177     +3249     
+ Partials      404       57      -347     

Flag	Coverage Δ
integrationtests	64.16% <ø> (-0.01%)	⬇️
unittests	86.55% <100.00%> (-0.06%)	⬇️

Components	Coverage Δ
pkg_aws_library	∅ <ø> (∅)
pkg_celery_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	76.69% <ø> (-8.25%)	⬇️
agent	∅ <ø> (∅)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	78.10% <ø> (-12.80%)	⬇️
dynamic_scheduler	∅ <ø> (∅)
dynamic_sidecar	81.87% <ø> (-8.56%)	⬇️
efs_guardian	∅ <ø> (∅)
invitations	∅ <ø> (∅)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	86.55% <100.00%> (+0.09%)	⬆️
webclient	∅ <ø> (∅)
webserver	59.15% <ø> (-28.55%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 699808f...5d27ec8. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 5d27ec8.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	system-tests	Base branch is broken, but retries were needed. Could be early signs of flakiness 👀		2	View	View

[pcrespov]

thx. Left some suggestions

[bisgaard-itis]

@Mergifyio queue

[mergify]

queue

🟠 Waiting for conditions to match

• -closed [📌 queue requirement]
• any of: [🔀 queue conditions]
  • all of: [📌 queue conditions of queue default]
    • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
    • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
    • #approved-reviews-by>=2
    • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
    • #changes-requested-reviews-by=0
    • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
    • #review-threads-unresolved=0
    • -conflict
    • -draft
    • base=master
    • label!=🤖-do-not-merge
    • label=🤖-automerge
    • any of: [🛡 GitHub branch protection]
      • check-skipped = deploy to dockerhub
      • check-neutral = deploy to dockerhub
      • check-success = deploy to dockerhub
    • any of: [🛡 GitHub branch protection]
      • check-success = system-tests
      • check-neutral = system-tests
      • check-skipped = system-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = unit-tests
      • check-neutral = unit-tests
      • check-skipped = unit-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = check OAS' are up to date
      • check-neutral = check OAS' are up to date
      • check-skipped = check OAS' are up to date
    • any of: [🛡 GitHub branch protection]
      • check-success = integration-tests
      • check-neutral = integration-tests
      • check-skipped = integration-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = build-test-images (frontend) / build-test-images
      • check-neutral = build-test-images (frontend) / build-test-images
      • check-skipped = build-test-images (frontend) / build-test-images
    • any of: [🛡 GitHub branch protection]
      • check-success = SonarCloud Code Analysis
      • check-neutral = SonarCloud Code Analysis
      • check-skipped = SonarCloud Code Analysis
• -conflict [📌 queue requirement]
• -draft [📌 queue requirement]
• any of: [📌 queue -> configuration change requirements]
  • -mergify-configuration-changed
  • check-success = Configuration changed

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mrnicegyu11]

force merged upon request