Skip to content

♻️ Refactor Models, Schema Validation, and DB Utilities (Spin-off 1 from PR #8141) #8360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Sep 12, 2025
Merged

♻️ Refactor Models, Schema Validation, and DB Utilities (Spin-off 1 from PR #8141) #8360

merged 14 commits into from
Sep 12, 2025

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Sep 11, 2025 •
edited
Loading

What do these changes do?

This pull request is a spin-off from PR #8141, which focuses on refactoring the project.workbench and therefore includes many many changes. This PR can be merged independently and will help reduce the size of the original PR and therefore make it easier to review. The changes here are mostly enhancements and utilities and do not introduce critical modifications to existing features.

Model Validation and Schema Improvements

  • Refactored description and thumbnail fields in ProjectGet and BaseProjectModel to use Pydantic's BeforeValidator for handling None and empty strings, and removed legacy field validators for cleaner code
  • Changed progress field in Node from float | None to int | None for stricter type enforcement, and added a new required_resources field to Node.
  • Improved handling of deprecated fields in Node by marking them with deprecated=True in the field metadata.

Database Utility Enhancements

  • Added an exists method to ProjectsRepo for checking project existence in the database, using a new pass_or_acquire_connection helper.
  • Refactored node creation logic in ProjectNodesRepo.add to simplify value preparation and ensure correct model dumping, and added inputs_required to ProjectNodeCreate.
  • Added model_dump_as_node methods to both ProjectNodeCreate and ProjectNode for converting database models to API models, excluding database-specific fields.

Testing and Helper Improvements

  • Improved test coverage for service key and version factories, and added a new helper function assert_equal_ignoring_none for flexible dictionary comparison in tests.
  • Enhanced the test_models_projects_to_jobs test to handle default column values and JSON serialization for dictionary fields, ensuring compatibility with schema changes.

Type Alias and Import Cleanups

  • Standardized the definition of NodeID as a type alias for UUID, and cleaned up imports in several modules for clarity and correctness.

Related issue/s

How to test

cd packages/models-library
make install-dev
make tests

cd packages/postgres-database
make install-dev
make tests

Dev-ops

None

@pcrespov pcrespov self-assigned this Sep 11, 2025
@pcrespov pcrespov added a:database associated to postgres service and postgres-database package a:models-library t:maintenance Some planned maintenance work labels Sep 11, 2025
@codecov
Copy link

codecov bot commented Sep 11, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.90%. Comparing base (14fa3f4) to head (c6903cd).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #8360      +/-   ##
==========================================
+ Coverage   87.88%   87.90%   +0.01%     
==========================================
  Files        1944     1945       +1     
  Lines       75603    75620      +17     
  Branches     1316     1316              
==========================================
+ Hits        66447    66470      +23     
+ Misses       8759     8753       -6     
  Partials      397      397
Flag Coverage Δ
integrationtests 63.97% <0.00%> (+0.01%) ⬆️
unittests 86.56% <94.44%> (+<0.01%) ⬆️
Components Coverage Δ
pkg_aws_library 93.59% <ø> (ø)
pkg_celery_library 84.71% <ø> (ø)
pkg_dask_task_models_library 79.33% <ø> (ø)
pkg_models_library 93.09% <100.00%> (-0.01%) ⬇️
pkg_notifications_library 85.20% <ø> (ø)
pkg_postgres_database 87.95% <71.42%> (-0.08%) ⬇️
pkg_service_integration 70.19% <ø> (ø)
pkg_service_library 71.95% <ø> (ø)
pkg_settings_library 90.19% <ø> (ø)
pkg_simcore_sdk 85.15% <ø> (+0.11%) ⬆️
agent 93.53% <ø> (ø)
api_server 91.94% <ø> (-0.01%) ⬇️
autoscaling 95.77% <ø> (ø)
catalog 92.36% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 92.37% <ø> (ø)
datcore_adapter 97.94% <ø> (ø)
director 75.81% <ø> (ø)
director_v2 91.01% <ø> (+0.06%) ⬆️
dynamic_scheduler 96.27% <ø> (ø)
dynamic_sidecar 90.46% <ø> (ø)
efs_guardian 89.62% <ø> (ø)
invitations 91.44% <ø> (ø)
payments 92.61% <ø> (ø)
resource_usage_tracker 92.24% <ø> (+0.10%) ⬆️
storage 86.53% <ø> (-0.13%) ⬇️
webclient ∅ <ø> (∅)
webserver 87.96% <100.00%> (+0.02%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14fa3f4...c6903cd. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pcrespov pcrespov changed the title WIP:♻️ Is8141/workbench pr spinoff 1 (Part 1) ♻️ Refactor Models, Schema Validation, and DB Utilities (Spin-off 1 from PR #8141) Sep 11, 2025
@pcrespov pcrespov marked this pull request as ready for review September 11, 2025 20:41
@mergify
Copy link
Contributor

mergify bot commented Sep 11, 2025 •
edited
Loading

🧪 CI Insights

Here's what we observed from your CI run for c6903cd.

❌ Job Failures

Pipeline Job Health on master Retries 🔍 CI Insights 📄 Logs
PR CI check OAS' are up to date Unknown 0 View View

✅ Passed Jobs With Interesting Signals

Pipeline Job Signal Health on master Retries 🔍 CI Insights 📄 Logs
CI system-tests Base branch is broken, but the job passed. Looks like this might be a real fix 💪 Broken 0 View View
unit-tests Base branch is broken, but retries were needed. Could be early signs of flakiness 👀 Broken 1 View View

sanderegg
sanderegg approved these changes Sep 12, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added a few comments.

wvangeit
wvangeit approved these changes Sep 12, 2025
View reviewed changes
Copy link
Contributor

@wvangeit wvangeit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

GitHK
GitHK approved these changes Sep 12, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look at my concerns

@pcrespov pcrespov added this to the Cheops milestone Sep 12, 2025
giancarloromeo
giancarloromeo approved these changes Sep 12, 2025
View reviewed changes
Copy link
Contributor

@giancarloromeo giancarloromeo left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We reviewed offline

@pcrespov pcrespov enabled auto-merge (squash) September 12, 2025 16:53
@pcrespov pcrespov added the 🤖-automerge marks PR as ready to be merged for Mergify label Sep 12, 2025
@pcrespov
Copy link
Member Author

@mergify queue

@mergify
Copy link
Contributor

mergify bot commented Sep 12, 2025 •
edited
Loading

queue

🟠 Waiting for conditions to match

  • -closed [📌 queue requirement]
  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
      • any of: [🛡 GitHub branch protection]
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
        • check-success = check OAS' are up to date
      • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • label!=🤖-do-not-merge
      • label=🤖-automerge
      • any of: [🛡 GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = build-test-images (frontend) / build-test-images
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
      • any of: [🛡 GitHub branch protection]
        • check-success = SonarCloud Code Analysis
        • check-neutral = SonarCloud Code Analysis
        • check-skipped = SonarCloud Code Analysis
  • -conflict [📌 queue requirement]
  • -draft [📌 queue requirement]
  • any of: [📌 queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed

@sonarqubecloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@pcrespov
Copy link
Member Author

I created #8365 to follow up on the fields marked as deprecated in the Node model.

@pcrespov pcrespov disabled auto-merge September 12, 2025 19:19
@pcrespov pcrespov merged commit 87820ae into ITISFoundation:master Sep 12, 2025
144 of 148 checks passed
@pcrespov pcrespov deleted the is8141/workbench-pr-spinoff-1 branch September 12, 2025 19:19
@pcrespov pcrespov mentioned this pull request Sep 15, 2025
Merged
This was referenced Sep 18, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@giancarloromeo giancarloromeo giancarloromeo approved these changes

@wvangeit wvangeit wvangeit approved these changes

@GitHK GitHK GitHK approved these changes

@sanderegg sanderegg sanderegg approved these changes

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66 matusdrobuliak66 is a code owner

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis bisgaard-itis is a code owner

Assignees

@pcrespov pcrespov

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:database associated to postgres service and postgres-database package a:models-library t:maintenance Some planned maintenance work

Projects

None yet

Milestone

Cheops

Development

Successfully merging this pull request may close these issues.

5 participants

@pcrespov @giancarloromeo @wvangeit @GitHK @sanderegg