Skip to content

🎨 Run each job creation as a single task in api-worker #8306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Sep 4, 2025
Merged

🎨 Run each job creation as a single task in api-worker #8306

merged 13 commits into from
Sep 4, 2025

Conversation

@bisgaard-itis
Copy link
Contributor

@bisgaard-itis bisgaard-itis commented Sep 4, 2025 •
edited
Loading

What do these changes do?

  • When using the map endpoint in the api-server we now create a separate celery task for each job creation.
  • Enhance tests to check that multiple celery tasks are created and job order is preserved by map endpoint

Related issue/s

How to test

Run

python -m pytest tests/unit/api_functions/celery -vv

from services/api-server

Dev-ops

@bisgaard-itis bisgaard-itis self-assigned this Sep 4, 2025
@bisgaard-itis bisgaard-itis added this to the Cheops milestone Sep 4, 2025
@bisgaard-itis bisgaard-itis marked this pull request as ready for review September 4, 2025 10:25
@codecov
Copy link

codecov bot commented Sep 4, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.00%. Comparing base (ce96a9b) to head (b3298e3).
⚠️ Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (ce96a9b) and HEAD (b3298e3). Click for more details.

HEAD has 31 uploads less than BASE
Flag BASE (ce96a9b) HEAD (b3298e3)
unittests 32 1
Additional details and impacted files 
@@             Coverage Diff             @@
##           master    #8306       +/-   ##
===========================================
- Coverage   87.85%   69.00%   -18.85%     
===========================================
  Files        1940      875     -1065     
  Lines       75342    39263    -36079     
  Branches     1311      175     -1136     
===========================================
- Hits        66189    27095    -39094     
- Misses       8758    12111     +3353     
+ Partials      395       57      -338
Flag Coverage Δ
integrationtests 64.04% <ø> (-0.05%) ⬇️
unittests 91.91% <100.00%> (+5.40%) ⬆️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 76.90% <ø> (-8.08%) ⬇️
agent ∅ <ø> (∅)
api_server 91.91% <100.00%> (-0.03%) ⬇️
autoscaling ∅ <ø> (∅)
catalog ∅ <ø> (∅)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 78.09% <ø> (-13.02%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 81.87% <ø> (-8.59%) ⬇️
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage ∅ <ø> (∅)
webclient ∅ <ø> (∅)
webserver 58.91% <ø> (-29.06%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce96a9b...b3298e3. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mergify
Copy link
Contributor

mergify bot commented Sep 4, 2025 •
edited
Loading

🧪 CI Insights

Here's what we observed from your CI run for b3298e3.

🟢 All jobs passed!

But CI Insights is watching 👀

pcrespov
pcrespov approved these changes Sep 4, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

wvangeit
wvangeit approved these changes Sep 4, 2025
View reviewed changes
Copy link
Contributor

@wvangeit wvangeit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thx.

@bisgaard-itis bisgaard-itis enabled auto-merge (squash) September 4, 2025 12:26
@bisgaard-itis
Copy link
Contributor Author

bisgaard-itis commented Sep 4, 2025

@Mergifyio queue

@bisgaard-itis bisgaard-itis added the 🤖-automerge marks PR as ready to be merged for Mergify label Sep 4, 2025
@mergify
Copy link
Contributor

mergify bot commented Sep 4, 2025 •
edited
Loading

queue

🛑 Configuration not compatible with a branch protection setting

The branch protection setting Require branches to be up to date before merging is not compatible with max_parallel_checks>1, queue_conditions != merge_conditions and must be unset.

@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 4, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@bisgaard-itis bisgaard-itis merged commit 95dde06 into ITISFoundation:master Sep 4, 2025
91 of 95 checks passed
@pcrespov pcrespov mentioned this pull request Sep 11, 2025
Closed
16 tasks
This was referenced Sep 18, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wvangeit wvangeit wvangeit approved these changes

@pcrespov pcrespov pcrespov approved these changes

Assignees

@bisgaard-itis bisgaard-itis

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:apiserver api-server service a:celery-library

Projects

None yet

Milestone

Cheops

Development

Successfully merging this pull request may close these issues.

3 participants

@bisgaard-itis @wvangeit @pcrespov