Skip to content

🎨 Introduce usage of 🛞 find tools to replace standard linux find, and some UV tweaks (🚨) #8088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Jul 10, 2025
Merged

🎨 Introduce usage of 🛞 find tools to replace standard linux find, and some UV tweaks (🚨) #8088

merged 14 commits into from
Jul 10, 2025

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Jul 9, 2025 •
edited
Loading

What do these changes do?

This PR introduces Rust fd-find which is a Rust-based replacement for the usual ubuntu find tool. Advantages are:

This PR also tweaks how Dockerfile are setup with regard to UV, following https://docs.astral.sh/uv/guides/integration/docker/#caching and https://docs.astral.sh/uv/guides/integration/docker/#compiling-bytecode

All these changes should boost startup times of prod/devel containers.

Related issue/s

How to test

Dev-ops

@sanderegg sanderegg added this to the Engage milestone Jul 9, 2025
@sanderegg sanderegg self-assigned this Jul 9, 2025
@sanderegg sanderegg added the t:maintenance Some planned maintenance work label Jul 9, 2025
@sanderegg sanderegg force-pushed the maintenance/introduce-rust-find-and-uv-recommendations branch from 9d4e6a2 to 382ce9e Compare July 9, 2025 15:09
pcrespov
pcrespov approved these changes Jul 9, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx. Very promising!

Q: why done we have a compose mechanism in place for the Dockerfiles so we can have some template/base Dockerfiles and easily change them everywhere.

Same goes for the entrypoints. Perhaps we should have like a small shell-like shared library that gets installed and the entrypoints could call these common functions
image

@codecov
Copy link

codecov bot commented Jul 9, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.06%. Comparing base (601c3ef) to head (5e7554f).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #8088      +/-   ##
==========================================
+ Coverage   88.26%   90.06%   +1.79%     
==========================================
  Files        1864     1583     -281     
  Lines       71869    62663    -9206     
  Branches     1264      626     -638     
==========================================
- Hits        63433    56435    -6998     
+ Misses       8069     6063    -2006     
+ Partials      367      165     -202
Flag Coverage Δ
integrationtests 64.23% <ø> (-0.05%) ⬇️
unittests 88.47% <ø> (+1.60%) ⬆️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library 93.24% <ø> (ø)
pkg_notifications_library 85.26% <ø> (ø)
pkg_postgres_database 88.22% <ø> (ø)
pkg_service_integration 69.92% <ø> (ø)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.05% <ø> (ø)
agent 96.29% <ø> (ø)
api_server 92.83% <ø> (ø)
autoscaling 95.88% <ø> (ø)
catalog 92.58% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 92.35% <ø> (+0.56%) ⬆️
datcore_adapter 97.94% <ø> (ø)
director 76.77% <ø> (ø)
director_v2 91.02% <ø> (-0.06%) ⬇️
dynamic_scheduler 96.69% <ø> (ø)
dynamic_sidecar 90.09% <ø> (ø)
efs_guardian 89.65% <ø> (ø)
invitations 93.60% <ø> (ø)
payments 92.57% <ø> (ø)
resource_usage_tracker 92.25% <ø> (+0.16%) ⬆️
storage 86.51% <ø> (+0.12%) ⬆️
webclient ∅ <ø> (∅)
webserver 88.59% <ø> (-0.03%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 601c3ef...5e7554f. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg requested a review from Copilot July 9, 2025 15:19
@sanderegg
Copy link
Member Author

sanderegg commented Jul 9, 2025

thx. Very promising!

Q: why done we have a compose mechanism in place for the Dockerfiles so we can have some template/base Dockerfiles and easily change them everywhere.

Same goes for the entrypoints. Perhaps we should have like a small shell-like shared library that gets installed and the entrypoints could call these common functions image

@pcrespov I think what we do now is a bit contraproductive.
Why build 15 times more or less the same Dockerfile?
we should actually build a base (or multiple) base images, and then just pull from it to build all the services.

This comment was marked as outdated.

Sign in to view

@sanderegg sanderegg requested a review from Copilot July 9, 2025 15:45
Copilot AI reviewed Jul 9, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR replaces traditional find commands with the Rust-based fd-find (fdfind) tool across service entrypoints and boot scripts for faster file ownership operations, and centralizes Ultraviolet (UV) bytecode compilation flags in Dockerfiles.

  • Swap find invocations for fdfind in entrypoint.sh and boot.sh of all services
  • Add fd-find to apt-get install in each service’s Dockerfile
  • Move UV bytecode compilation environment variables into the appropriate build stages and remove duplicates

Reviewed Changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated 3 comments.

File Pattern Description
services/*/docker/entrypoint.sh Replaced find with fdfind for changing file ownership in development mode
services/*/docker/boot.sh Swapped find for fdfind to generate --reload-dir arguments in debug mode
**/Dockerfile Installed fd-find package and consolidated UV_COMPILE_BYTECODE & UV_LINK_MODE env vars

@sanderegg sanderegg added the 🤖-automerge marks PR as ready to be merged for Mergify label Jul 9, 2025
@sanderegg
Copy link
Member Author

sanderegg commented Jul 9, 2025

@mergify queue

@mergify
Copy link
Contributor

mergify bot commented Jul 9, 2025 •
edited
Loading

queue

🟠 Waiting for conditions to match

  • -closed [📌 queue requirement]
  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
      • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • label!=🤖-do-not-merge
      • label=🤖-automerge
      • any of: [🛡 GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = check OAS' are up to date
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
      • any of: [🛡 GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = build-test-images (frontend) / build-test-images
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
  • -conflict [📌 queue requirement]
  • -draft [📌 queue requirement]
  • any of: [📌 queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed

@sanderegg sanderegg requested a review from odeimaiz July 10, 2025 05:57
@sanderegg sanderegg force-pushed the maintenance/introduce-rust-find-and-uv-recommendations branch from 4def564 to 5e7554f Compare July 10, 2025 06:39
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 10, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

odeimaiz
odeimaiz reviewed Jul 10, 2025
View reviewed changes
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@sanderegg sanderegg merged commit e1d722b into ITISFoundation:master Jul 10, 2025
95 checks passed
@sanderegg sanderegg deleted the maintenance/introduce-rust-find-and-uv-recommendations branch July 10, 2025 08:00
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@odeimaiz odeimaiz odeimaiz left review comments

Copilot code review Copilot Copilot left review comments

@giancarloromeo giancarloromeo giancarloromeo approved these changes

@GitHK GitHK GitHK approved these changes

@pcrespov pcrespov pcrespov approved these changes

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66 matusdrobuliak66 is a code owner

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis bisgaard-itis is a code owner

Assignees

@sanderegg sanderegg

Labels

🤖-automerge marks PR as ready to be merged for Mergify t:maintenance Some planned maintenance work

Projects

None yet

Milestone

Engage

Development

Successfully merging this pull request may close these issues.

5 participants

@sanderegg @giancarloromeo @GitHK @pcrespov @odeimaiz