Skip to content

🎨 introduce comp_run_snapshot_tasks table (πŸ—ƒοΈ) 🚨 #7758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 31 commits into from
May 30, 2025
Merged

🎨 introduce comp_run_snapshot_tasks table (πŸ—ƒοΈ) 🚨 #7758

merged 31 commits into from
May 30, 2025

Conversation

@matusdrobuliak66
Copy link
Collaborator

@matusdrobuliak66 matusdrobuliak66 commented May 26, 2025 •
edited
Loading

What do these changes do?

  • πŸ—ƒοΈ Added a new comp_run_snapshot_tasks table to store the task data for each individual run. The existing comp_tasks table only caches the latest run and handles data differently, as it also includes dynamic and frontend services. Historically, this setup predates the projects and then project_nodes tables.
  • πŸ—ƒοΈ Added a dag_adjacency_list field to the comp_runs table. Previously, this information was stored in the comp_pipeline table, which was only cached for the latest run. Now, we persist the DAG structure for every run individually.

Related issue/s

How to test

Dev-ops

@codecov
Copy link

codecov bot commented May 26, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 81.41593% with 21 lines in your changes missing coverage. Please review.

Project coverage is 88.63%. Comparing base (125e103) to head (5ff7d08).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7758      +/-   ##
==========================================
+ Coverage   86.78%   88.63%   +1.85%     
==========================================
  Files        1841     1553     -288     
  Lines       71507    61626    -9881     
  Branches     1215      546     -669     
==========================================
- Hits        62060    54625    -7435     
+ Misses       9105     6857    -2248     
+ Partials      342      144     -198
Flag Coverage Ξ”
integrationtests 64.37% <72.05%> (-0.07%) ⬇️
unittests 88.39% <80.53%> (+1.84%) ⬆️
Components Coverage Ξ”
api βˆ… <ΓΈ> (βˆ…)
pkg_aws_library βˆ… <ΓΈ> (βˆ…)
pkg_dask_task_models_library βˆ… <ΓΈ> (βˆ…)
pkg_models_library 93.08% <ΓΈ> (ΓΈ)
pkg_notifications_library 85.26% <ΓΈ> (ΓΈ)
pkg_postgres_database 88.07% <64.44%> (-0.55%) ⬇️
pkg_service_integration βˆ… <ΓΈ> (βˆ…)
pkg_service_library βˆ… <ΓΈ> (βˆ…)
pkg_settings_library βˆ… <ΓΈ> (βˆ…)
pkg_simcore_sdk 84.97% <90.00%> (+0.08%) ⬆️
agent 96.29% <ΓΈ> (ΓΈ)
api_server 91.77% <ΓΈ> (ΓΈ)
autoscaling 96.03% <ΓΈ> (ΓΈ)
catalog 92.25% <ΓΈ> (ΓΈ)
clusters_keeper 99.13% <ΓΈ> (ΓΈ)
dask_sidecar 91.67% <ΓΈ> (ΓΈ)
datcore_adapter 97.94% <ΓΈ> (ΓΈ)
director 76.73% <ΓΈ> (ΓΈ)
director_v2 91.07% <93.10%> (+0.07%) ⬆️
dynamic_scheduler 96.69% <ΓΈ> (ΓΈ)
dynamic_sidecar 90.14% <ΓΈ> (ΓΈ)
efs_guardian 89.65% <ΓΈ> (ΓΈ)
invitations 93.00% <ΓΈ> (ΓΈ)
payments 92.57% <ΓΈ> (ΓΈ)
resource_usage_tracker 89.09% <ΓΈ> (+0.10%) ⬆️
storage 87.53% <ΓΈ> (ΓΈ)
webclient βˆ… <ΓΈ> (βˆ…)
webserver 83.94% <ΓΈ> (ΓΈ)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Ξ” = absolute <relative> (impact), ΓΈ = not affected, ? = missing data
Powered by Codecov. Last update 125e103...5ff7d08. Read the comment docs.

πŸš€ New features to boost your workflow:
  • πŸ“¦ JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@matusdrobuliak66 matusdrobuliak66 self-assigned this May 26, 2025
@matusdrobuliak66 matusdrobuliak66 added the a:director-v2 issue related with the director-v2 service label May 27, 2025
@matusdrobuliak66 matusdrobuliak66 added this to the Bazinga! milestone May 27, 2025
sanderegg
sanderegg reviewed May 27, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! thanks a lot!
A few comments that we can discuss tomorrow morning.

@matusdrobuliak66 matusdrobuliak66 marked this pull request as ready for review May 28, 2025 09:32
GitHK
GitHK reviewed May 28, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: what will happen to the comp_runs table after this? will it still be there? Te dynamic-services require it

@matusdrobuliak66
Copy link
Collaborator Author

matusdrobuliak66 commented May 28, 2025

Q: what will happen to the comp_runs table after this? will it still be there? Te dynamic-services require it

Yes, all tables will still be there.

@sanderegg
Copy link
Member

sanderegg commented May 28, 2025

Q: what will happen to the comp_runs table after this? will it still be there? Te dynamic-services require it

@GitHK you do not require comp_runs but comp_tasks

sanderegg
sanderegg approved these changes May 28, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good! Thanks for all the changes!

pcrespov
pcrespov approved these changes May 28, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx for the explanations offline.
I left some minor suggestions

I would really like to add tests to the utls_ in the postgres. I am happy to show you how to do that in a very fast way

@matusdrobuliak66 matusdrobuliak66 enabled auto-merge (squash) May 29, 2025 16:21
@matusdrobuliak66
Copy link
Collaborator Author

matusdrobuliak66 commented May 29, 2025

@mergify queue

@matusdrobuliak66 matusdrobuliak66 disabled auto-merge May 29, 2025 16:38
@ITISFoundation ITISFoundation deleted a comment from mergify bot May 29, 2025
@mergify
Copy link
Contributor

mergify bot commented May 29, 2025 •
edited
Loading

queue

🟠 Waiting for conditions to match

  • -closed [πŸ“Œ queue requirement]
  • any of: [πŸ”€ queue conditions]
    • all of: [πŸ“Œ queue conditions of queue default]
      • label!=πŸ€–-do-not-merge
      • label=πŸ€–-automerge
      • #approved-reviews-by >= 2 [πŸ›‘ GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [πŸ›‘ GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [πŸ›‘ GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • branch-protection-review-decision = APPROVED [πŸ›‘ GitHub branch protection]
      • any of: [πŸ›‘ GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [πŸ›‘ GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [πŸ›‘ GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [πŸ›‘ GitHub branch protection]
        • check-success = check OAS' are up to date
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
      • any of: [πŸ›‘ GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
      • any of: [πŸ›‘ GitHub branch protection]
        • check-success = build-test-images (frontend) / build-test-images
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
  • -conflict [πŸ“Œ queue requirement]
  • -draft [πŸ“Œ queue requirement]
  • any of: [πŸ“Œ queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed

@matusdrobuliak66 matusdrobuliak66 added the πŸ€–-do-not-merge (optional) blocks Mergify from merging the PR label May 29, 2025
@sonarqubecloud
Copy link

sonarqubecloud bot commented May 29, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@matusdrobuliak66 matusdrobuliak66 merged commit c2cf892 into ITISFoundation:master May 30, 2025
94 of 95 checks passed
@matusdrobuliak66 matusdrobuliak66 changed the title 🎨 introduce comp_run_snapshot_tasks table (πŸ—ƒοΈ) 🎨 introduce comp_run_snapshot_tasks table (πŸ—ƒοΈ) 🚨 May 30, 2025
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Jun 6, 2025
Closed
92 tasks
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@pcrespov pcrespov pcrespov approved these changes

@sanderegg sanderegg sanderegg approved these changes

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis

@GitHK GitHK Awaiting requested review from GitHK GitHK is a code owner

Assignees

@matusdrobuliak66 matusdrobuliak66

Labels

πŸ€–-do-not-merge (optional) blocks Mergify from merging the PR a:director-v2 issue related with the director-v2 service

Projects

None yet

Milestone

Bazinga!

Development

Successfully merging this pull request may close these issues.

Add comp_runs_nodes table to director-v2

4 participants

@matusdrobuliak66 @sanderegg @GitHK @pcrespov