Skip to content

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Jun 27, 2025

Context

Running the CI and looking at the workflow file is not a nice experience. This PR is a first step towards rationalizing the CI runs.
This PR modularizes towards simpler CI workflow file with a longer term goal to have maybe faster CI feedback. Ideas currently around are:

  • separate workflows for unit tests/integration tests/system tests that could have dependencies
  • draft PRs could for example only run unit tests and non draft also the integration/system tests to have faster feedback
  • other ideas??

This PR should not have any effect for now in terms of functionality but more in terms of sustainability

@sanderegg sanderegg added this to the Engage milestone Jun 27, 2025
@sanderegg sanderegg self-assigned this Jun 27, 2025
@sanderegg sanderegg added a:infra+ops maintenance of infrastructure or operations (discussed in retro) t:maintenance Some planned maintenance work labels Jun 27, 2025
@codecov
Copy link

codecov bot commented Jun 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.87%. Comparing base (2d1f459) to head (ce70e5a).
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8001      +/-   ##
==========================================
- Coverage   87.88%   87.87%   -0.02%     
==========================================
  Files        1853     1853              
  Lines       71478    71478              
  Branches     1258     1258              
==========================================
- Hits        62822    62814       -8     
- Misses       8292     8300       +8     
  Partials      364      364              
Flag Coverage Δ *Carryforward flag
integrationtests 64.24% <ø> (-0.02%) ⬇️ Carriedforward from 2f1b484
unittests 86.49% <ø> (-0.02%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

Components Coverage Δ
api 76.84% <ø> (ø)
pkg_aws_library 93.93% <ø> (ø)
pkg_celery_library 87.15% <ø> (ø)
pkg_dask_task_models_library 79.62% <ø> (ø)
pkg_models_library 93.27% <ø> (ø)
pkg_notifications_library 85.26% <ø> (ø)
pkg_postgres_database 88.19% <ø> (ø)
pkg_service_integration 69.92% <ø> (ø)
pkg_service_library 71.63% <ø> (ø)
pkg_settings_library 90.64% <ø> (ø)
pkg_simcore_sdk 85.05% <ø> (-0.06%) ⬇️
agent 96.29% <ø> (ø)
api_server 92.64% <ø> (ø)
autoscaling 96.03% <ø> (ø)
catalog 92.29% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 91.79% <ø> (-0.57%) ⬇️
datcore_adapter 97.94% <ø> (ø)
director 76.73% <ø> (ø)
director_v2 91.03% <ø> (ø)
dynamic_scheduler 96.69% <ø> (ø)
dynamic_sidecar 90.09% <ø> (ø)
efs_guardian 89.65% <ø> (ø)
invitations 93.60% <ø> (ø)
payments 92.57% <ø> (ø)
resource_usage_tracker 89.00% <ø> (-0.06%) ⬇️
storage 86.44% <ø> (+0.04%) ⬆️
webclient ∅ <ø> (∅)
webserver 87.64% <ø> (-0.01%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d1f459...ce70e5a. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg marked this pull request as ready for review June 27, 2025 13:08
@sanderegg sanderegg requested a review from pcrespov as a code owner June 27, 2025 13:08
@sanderegg sanderegg added the 🤖-automerge marks PR as ready to be merged for Mergify label Jun 27, 2025
@sanderegg
Copy link
Member Author

@mergify queue

@mergify
Copy link
Contributor

mergify bot commented Jun 27, 2025

queue

🛑 The pull request has been removed from the queue default

The following conditions don't match anymore:

  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • any of: [🛡 GitHub branch protection]
        • check-neutral = deploy to dockerhub
        • check-skipped = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-neutral = unit-tests
        • check-skipped = unit-tests
        • check-success = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
        • check-success = build-test-images (frontend) / build-test-images

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the CI workflow to set up a common environment for SimCore jobs and adds a new debugging target for mypy. Key changes include:

  • Introduction of a new “mypy-debug” target in the Makefile.
  • Refactoring of numerous CI workflow files to use a common composite action for environment setup.
  • Addition of new composite actions for setting up the SimCore environment and for downloading and loading Docker images.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
scripts/common.Makefile Added new target “mypy-debug” for enhanced mypy profiling.
mypy.ini Minor formatting adjustment.
.github/workflows/ci-testing-deploy.yml Updated multiple jobs to use “Setup environment” via a composite action.
.github/workflows/_reusable-build-images.yml Refactored to use the common environment setup action.
.github/actions/setup-simcore-env/action.yml Added a new composite action for environment setup.
.github/actions/download-load-docker-images/action.yml Introduced a composite action for downloading and loading Docker images.

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love it! thx!

  • the idea of running drafts differently is also very nice!

@sonarqubecloud
Copy link

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2025

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

@sanderegg
Copy link
Member Author

@Mergifyio requeue

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2025

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request.

@mergify
Copy link
Contributor

mergify bot commented Jun 28, 2025

queue

🟠 Waiting for conditions to match

  • -closed [📌 queue requirement]
  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • any of: [🛡 GitHub branch protection]
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
        • check-success = build-test-images (frontend) / build-test-images
      • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
      • label!=🤖-do-not-merge
      • label=🤖-automerge
      • any of: [🛡 GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = check OAS' are up to date
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
      • any of: [🛡 GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
  • -conflict [📌 queue requirement]
  • -draft [📌 queue requirement]
  • any of: [📌 queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed

@sanderegg sanderegg merged commit 7f28bbc into ITISFoundation:master Jun 30, 2025
97 of 98 checks passed
@sanderegg sanderegg deleted the maintenance/ci-modularizing-1st-step branch June 30, 2025 04:42
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:infra+ops maintenance of infrastructure or operations (discussed in retro) t:maintenance Some planned maintenance work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants