Skip to content

🎨 Add removal of project documents from the Redis (Garbage Collection background task) #8177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 28 commits into from
Jul 31, 2025
Merged

🎨 Add removal of project documents from the Redis (Garbage Collection background task) #8177

merged 28 commits into from
Jul 31, 2025

Conversation

@matusdrobuliak66
Copy link
Collaborator

@matusdrobuliak66 matusdrobuliak66 commented Jul 30, 2025 •
edited
Loading

What do these changes do?

  • 🎨 Add removal of project documents from the Redis (Garbage Collection background task)

Related issue/s

How to test

osparc-simcore/services/web/server/tests/unit/with_dbs/04/garbage_collector/test_projects_document_service.py

Dev-ops

matusdrobuliak66 and others added 20 commits July 29, 2025 17:11
@matusdrobuliak66
Adds client session ID propagation for user actions
b2d3396 
Introduces client session ID to APIs and project update notifications,
enabling session-aware updates and improved multi-tab/user experience.

Allows the frontend to distinguish between updates from different
user sessions, preventing redundant UI refreshes for the originating
session when optimistic updates are used.

Relates to collaborative editing and real-time notification enhancements.
@matusdrobuliak66
Merge branch 'master' into is1647/collaboration-features-2
92cb19f
@matusdrobuliak66
Removes unused frontend outputs update logic
8c5b95d 
Cleans up obsolete code related to updating frontend node outputs,
streamlining the module and reducing maintenance overhead. This helps
avoid confusion and ensures only relevant logic is retained.
@matusdrobuliak66
fix tests
a68c91e
@matusdrobuliak66
Refactors project document update for clarity and atomicity
4d3d978 
Streamlines the workflow for updating project documents by separating
workbench updates from document and version increment logic. Improves
atomicity and clarity in concurrent operations, ensuring that project
documents and their versions remain consistent. Enhances maintainability
and reduces potential race conditions during project modifications.
@matusdrobuliak66
fix tests
503ef0f
@matusdrobuliak66
Merge branch 'master' into is1647/collaboration-features-2
f58843c
@matusdrobuliak66
review @sanderegg
ae5be6f
@matusdrobuliak66
review @sanderegg @pcrespov
0d5a6c3
@matusdrobuliak66
review @pcrespov
89a9394
@matusdrobuliak66
fix
ac80fed
@matusdrobuliak66
fix
26ac23b
@matusdrobuliak66
fix
0b8c698
@matusdrobuliak66
additional tests
e4b4a0b
@matusdrobuliak66
Adds periodic cleanup task for project documents
22cae1b 
Introduces a background job that regularly prunes project documents from Redis,
improving resource management and preventing data accumulation.
Integrates the new cleanup routine into the existing garbage collector plugin.
@matusdrobuliak66
Removes project documents with no active users
04110d4 
Introduces an admin cleanup function that scans project documents
in the Redis database and deletes those associated with projects
that do not have any connected users via socketio. Helps reduce
unnecessary storage and maintains data consistency.
@matusdrobuliak66
Refactors project document cleanup to use shared registry utility
ed21eb0 
Simplifies logic for identifying opened projects by introducing a shared utility function.
Improves reliability of project document removal by ensuring active projects are preserved.
Adds comprehensive unit tests to verify correct behavior in various edge cases.
@matusdrobuliak66
additional tests
b07870a
@matusdrobuliak66
Refactors project document removal unit tests with fixtures
ff0062f 
Improves test clarity and maintainability by introducing fixtures for Redis client, SocketIO server, and project document setup.
Simplifies test logic, enhances resource cleanup, and ensures more robust test isolation for project document removal scenarios.
@matusdrobuliak66
additional tests
998ff14
@codecov
Copy link

codecov bot commented Jul 30, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 95.23810% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.11%. Comparing base (2c424ad) to head (94c1cdc).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #8177      +/-   ##
==========================================
- Coverage   88.06%   85.11%   -2.96%     
==========================================
  Files        1901      739    -1162     
  Lines       73093    33838   -39255     
  Branches     1280      176    -1104     
==========================================
- Hits        64368    28800   -35568     
+ Misses       8345     4980    -3365     
+ Partials      380       58     -322
Flag Coverage Δ
integrationtests 64.13% <69.04%> (+0.01%) ⬆️
unittests 87.38% <95.23%> (+0.67%) ⬆️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 76.92% <ø> (-8.13%) ⬇️
agent ∅ <ø> (∅)
api_server ∅ <ø> (∅)
autoscaling ∅ <ø> (∅)
catalog ∅ <ø> (∅)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 77.06% <ø> (-13.94%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 87.31% <ø> (-2.77%) ⬇️
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage ∅ <ø> (∅)
webclient ∅ <ø> (∅)
webserver 88.15% <95.23%> (+0.04%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c424ad...94c1cdc. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@matusdrobuliak66 matusdrobuliak66 self-assigned this Jul 30, 2025
@matusdrobuliak66 matusdrobuliak66 added the a:webserver webserver's codebase. Assigning the area is particularly useful for bugs label Jul 30, 2025
@matusdrobuliak66 matusdrobuliak66 added this to the Engage milestone Jul 30, 2025
@matusdrobuliak66 matusdrobuliak66 marked this pull request as ready for review July 30, 2025 14:58
pcrespov
pcrespov approved these changes Jul 30, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx! left some suggestions

odeimaiz
odeimaiz approved these changes Jul 30, 2025
View reviewed changes
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merci

GitHK
GitHK reviewed Jul 31, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced by the approach, but I might be missing something. Please see my comment

sanderegg
sanderegg approved these changes Jul 31, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, thanks!

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 31, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@matusdrobuliak66 matusdrobuliak66 enabled auto-merge (squash) July 31, 2025 06:55
@matusdrobuliak66 matusdrobuliak66 added the 🤖-automerge marks PR as ready to be merged for Mergify label Jul 31, 2025
@matusdrobuliak66
Copy link
Collaborator Author

matusdrobuliak66 commented Jul 31, 2025

@mergify queue

@mergify
Copy link
Contributor

mergify bot commented Jul 31, 2025 •
edited
Loading

queue

🟠 Waiting for conditions to match

  • -closed [📌 queue requirement]
  • -conflict [📌 queue requirement]
  • -draft [📌 queue requirement]
  • any of: [📌 queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed
  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
      • label!=🤖-do-not-merge
      • label=🤖-automerge
      • any of: [🛡 GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = check OAS' are up to date
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
      • any of: [🛡 GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = build-test-images (frontend) / build-test-images
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
      • any of: [🛡 GitHub branch protection]
        • check-success = SonarCloud Code Analysis
        • check-neutral = SonarCloud Code Analysis
        • check-skipped = SonarCloud Code Analysis

@matusdrobuliak66 matusdrobuliak66 merged commit bca1a27 into ITISFoundation:master Jul 31, 2025
94 of 96 checks passed
This was referenced Aug 5, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GitHK GitHK GitHK left review comments

@pcrespov pcrespov pcrespov approved these changes

@odeimaiz odeimaiz odeimaiz approved these changes

@sanderegg sanderegg sanderegg approved these changes

@giancarloromeo giancarloromeo Awaiting requested review from giancarloromeo

Assignees

@matusdrobuliak66 matusdrobuliak66

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:webserver webserver's codebase. Assigning the area is particularly useful for bugs

Projects

None yet

Milestone

Engage

Development

Successfully merging this pull request may close these issues.

5 participants

@matusdrobuliak66 @GitHK @pcrespov @odeimaiz @sanderegg