♻️🚨 Improve DB query listing_projects performance (🗃️)

[matusdrobuliak66]

What do these changes do?

• ♻️ Improved the performance of the listing_projects database query (🗃️)
• Used EXPLAIN ANALYZE to analyze query performance. By adding indexes and refactoring the SQL query, many sequential scans were eliminated.

What was improved?

• Added a WHERE clause for user_groups in my_access_rights_subquery → Previously, it aggregated all project_to_groups data. With this change, user access rights are now determined through a join, eliminating the need to compare strings within a JSON array.
• Added .lateral() to access_rights_subquery → This helps the database engine avoid pre-aggregating all data in project_to_groups, processing only the projects that were already filtered (which, with an offset + limit combination, is usually just around 10 projects).
• Replaced FILTER with a WHERE clause → The FILTER condition prevented index scans from being used.
• Removed the projects_tags join → Tags are retrieved separately for each project later, making the join unnecessary at this stage.
• Optimized index usage → For example, indexing project type to prevent a sequential scan of all projects when listing templates.

What can still be improved?

• Remove total count computation → Currently, the total count is calculated for each listing request. Instead, the client can iterate until all data is retrieved.
• Remove the projects_to_products table and integrate its data into the projects table.

🚨 Double check user project permissions 🚨

Related issue/s

• https://git.speag.com/oSparc/osparc-infra/-/issues/67

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.16%. Comparing base (ef99643) to head (3af0640).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7475      +/-   ##
==========================================
+ Coverage   87.40%   89.16%   +1.76%     
==========================================
  Files        1737     1473     -264     
  Lines       67032    58116    -8916     
  Branches     1138      541     -597     
==========================================
- Hits        58589    51820    -6769     
+ Misses       8122     6146    -1976     
+ Partials      321      150     -171     

Flag	Coverage Δ
integrationtests	65.20% <20.00%> (+0.03%)	⬆️
unittests	88.23% <94.44%> (+1.64%)	⬆️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	91.94% <ø> (ø)
pkg_notifications_library	84.86% <ø> (ø)
pkg_postgres_database	88.18% <100.00%> (+<0.01%)	⬆️
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	85.46% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	90.83% <ø> (ø)
autoscaling	96.08% <ø> (ø)
catalog	91.82% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.29% <ø> (ø)
datcore_adapter	98.12% <ø> (ø)
director	76.87% <ø> (-0.10%)	⬇️
director_v2	91.30% <ø> (+0.02%)	⬆️
dynamic_scheduler	97.35% <ø> (ø)
dynamic_sidecar	90.11% <ø> (ø)
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <ø> (ø)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.23% <ø> (+0.10%)	⬆️
storage	87.17% <91.66%> (+0.08%)	⬆️
webclient	∅ <ø> (∅)
webserver	85.91% <100.00%> (+<0.01%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef99643...3af0640. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sanderegg]

very nice!
Thanks, how much faster does it get?

[odeimaiz]

🙏

[GitHK]

Nice, did you manage to test this on the production dataset to see if we actually get a boost in speed?

[matusdrobuliak66]

Nice, did you manage to test this on the production dataset to see if we actually get a boost in speed?

Tested in staging ... I expect 10x speedup

[matusdrobuliak66]

very nice! Thanks, how much faster does it get?

based on my experiments in staging 10x speedup

[pcrespov]

Great work ! 🎉 thx!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud