🎨Computational backend: improvements step 6

[sanderegg]

What do these changes do?

• remove distributed Variable that can timeout after 30 seconds and bring us in a undefined state and send tasks to the underlying dask backend without writing it to the DB
• removes concurrent access to the dask-scheduler as this defeats the purpose of limiting its access via a semaphore (capacity is 20, with concurrent access set to 10, we get potentially 200 concurrent calls that the dask-scheduler does not handle)
• removes concurrent access to the DB as many warnings showing DB access approach maximum were spotted.

Related issue/s

• relates to https://github.com/ITISFoundation/private-issues/issues/10

How to test

Dev-ops

[codecov]

Codecov Report

❌ Patch coverage is 80.64516% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.62%. Comparing base (a26142b) to head (2bde407).
⚠️ Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (a26142b) and HEAD (2bde407). Click for more details.

HEAD has 31 uploads less than BASE

Flag	BASE (a26142b)	HEAD (2bde407)
unittests	32	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #8397       +/-   ##
===========================================
- Coverage   87.90%   68.62%   -19.29%     
===========================================
  Files        1951      761     -1190     
  Lines       75951    35038    -40913     
  Branches     1336      175     -1161     
===========================================
- Hits        66763    24044    -42719     
- Misses       8788    10937     +2149     
+ Partials      400       57      -343     

Flag	Coverage Δ
integrationtests	63.92% <67.74%> (-0.04%)	⬇️
unittests	84.47% <74.19%> (-2.11%)	⬇️

Components	Coverage Δ
pkg_aws_library	∅ <ø> (∅)
pkg_celery_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	76.69% <ø> (-8.30%)	⬇️
agent	∅ <ø> (∅)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	90.91% <80.64%> (-0.06%)	⬇️
dynamic_scheduler	∅ <ø> (∅)
dynamic_sidecar	81.87% <ø> (-8.59%)	⬇️
efs_guardian	∅ <ø> (∅)
invitations	∅ <ø> (∅)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	∅ <ø> (∅)
webclient	∅ <ø> (∅)
webserver	58.78% <ø> (-29.20%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a26142b...2bde407. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 2bde407.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	integration-tests	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		0	View	View
	unit-tests	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		0	View	View

[Copilot]

Pull Request Overview

This PR removes the distributed.Variable mechanism from the Dask client module to address timeout issues that could lead to undefined states. The changes eliminate the use of distributed Variables for storing task futures and simplify the task processing pipeline by removing concurrent execution limits in favor of sequential processing.

• Removes distributed.Variable usage for storing and releasing task futures
• Replaces concurrent task processing with sequential execution to reduce scheduler load
• Removes unused imports and constants related to pipeline concurrency limits

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
dask_client.py	Removes distributed.Variable operations for task future storage and cleanup
_scheduler_dask.py	Converts concurrent task operations to sequential execution and adjusts concurrency limits
_scheduler_base.py	Replaces asyncio.gather with limited_gather for better concurrency control

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bisgaard-itis]

Thanks

[wvangeit]

Thanks