🐛On-demand computational backend failing

[sanderegg]

What do these changes do?

This pull request introduces several updates across multiple services to improve debugging workflows, enhance logging, and refine service configurations. The most significant changes include adding conditional logic for installing debugpy, updating Python debug options, reducing noisy loggers, and modifying service-specific configurations and the potential bugfix for on-demand clusters that were missing some ENV variables for the dask-scheduler due to the changes in .

Debugging Enhancements:

• Added conditional logic to check for the uv command before installing debugpy in debug mode across multiple services. This ensures compatibility with environments where uv is available. (services/agent/docker/boot.sh [1] services/api-server/docker/boot.sh [2] and others)
• Updated the Python debug command to include the -Xfrozen_modules=off option for better debugging support in services. (services/agent/docker/boot.sh [1] services/api-server/docker/boot.sh [2] and others)

Logging Improvements:

• Reduced verbosity of specific third-party loggers (e.g., aiobotocore, aio_pika) by dynamically adjusting their log levels based on the root logger's configuration. (services/clusters-keeper/src/simcore_service_clusters_keeper/core/application.py services/clusters-keeper/src/simcore_service_clusters_keeper/core/application.pyL26-R46)

Configuration Updates:

• Added shared volume and environment variable configurations for dask-sidecar in the docker-compose.yml file to support computational shared data. (services/clusters-keeper/src/simcore_service_clusters_keeper/data/docker-compose.yml [1] [2], a.k.a. the fix.

Related issue/s

How to test

Dev-ops

[Copilot]

Pull Request Overview

This PR improves debugging and logging workflows across multiple services and includes a critical fix for on-demand computational clusters by supplying missing environment variables and volume mounts for the Dask scheduler sidecar.

• Introduces shared data volume and environment variables for dask-sidecar to ensure proper scheduling on on-demand clusters
• Refactors get_scheduler_url for brevity and updates noise-reduction logic for third-party loggers
• Reorganizes imports and adjusts HTTP error response code for consistency

Reviewed Changes

Copilot reviewed 6 out of 22 changed files in this pull request and generated 4 comments.

File	Description
services/clusters-keeper/src/simcore_service_clusters_keeper/utils/dask.py	Inlined return in get_scheduler_url
services/clusters-keeper/src/simcore_service_clusters_keeper/data/docker-compose.yml	Added shared volume/env for dask-sidecar; updated LOG_LEVEL
services/clusters-keeper/src/simcore_service_clusters_keeper/core/application.py	Added noise-reduction for verbose loggers
packages/service-library/src/servicelib/aiohttp/rest_responses.py	Moved RESPONSE_MODEL_POLICY import; clarified comment

Files not reviewed (16)

• services/agent/docker/boot.sh: Language not supported
• services/api-server/docker/boot.sh: Language not supported
• services/autoscaling/docker/boot.sh: Language not supported
• services/catalog/docker/boot.sh: Language not supported
• services/clusters-keeper/docker/boot.sh: Language not supported
• services/dask-sidecar/docker/boot.sh: Language not supported
• services/datcore-adapter/docker/boot.sh: Language not supported
• services/director-v2/docker/boot.sh: Language not supported
• services/director/docker/boot.sh: Language not supported
• services/dynamic-scheduler/docker/boot.sh: Language not supported
• services/dynamic-sidecar/docker/boot.sh: Language not supported
• services/efs-guardian/docker/boot.sh: Language not supported
• services/invitations/docker/boot.sh: Language not supported
• services/notifications/docker/boot.sh: Language not supported
• services/payments/docker/boot.sh: Language not supported
• services/resource-usage-tracker/docker/boot.sh: Language not supported

Comments suppressed due to low confidence (1)

services/clusters-keeper/src/simcore_service_clusters_keeper/data/docker-compose.yml:6

• The volume computational_shared_data is referenced under the service but not declared in the top-level volumes: section, which will cause Docker Compose to fail. Please add a volumes: entry defining computational_shared_data:.

-      - computational_shared_data:${SIDECAR_COMP_SERVICES_SHARED_FOLDER:-/home/scu/computational_shared_data}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.82%. Comparing base (e8ce837) to head (e1e3e42).

❗ There is a different number of reports uploaded between BASE (e8ce837) and HEAD (e1e3e42). Click for more details.

HEAD has 31 uploads less than BASE

Flag	BASE (e8ce837)	HEAD (e1e3e42)
unittests	32	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #7735       +/-   ##
===========================================
- Coverage   87.37%   66.82%   -20.55%     
===========================================
  Files        1838      713     -1125     
  Lines       71156    33412    -37744     
  Branches     1201      174     -1027     
===========================================
- Hits        62173    22328    -39845     
- Misses       8653    11026     +2373     
+ Partials      330       58      -272     

Flag	Coverage Δ		*Carryforward flag
integrationtests	64.46% <ø> (ø)		Carriedforward from 15df825
unittests	93.28% <ø> (+6.68%)	⬆️

*This pull request uses carry forward flags. Click here to find out more.

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	∅ <ø> (∅)
pkg_dask_task_models_library	∅ <ø> (∅)
pkg_models_library	∅ <ø> (∅)
pkg_notifications_library	∅ <ø> (∅)
pkg_postgres_database	∅ <ø> (∅)
pkg_service_integration	∅ <ø> (∅)
pkg_service_library	∅ <ø> (∅)
pkg_settings_library	∅ <ø> (∅)
pkg_simcore_sdk	76.89% <ø> (-8.24%)	⬇️
agent	∅ <ø> (∅)
api_server	∅ <ø> (∅)
autoscaling	∅ <ø> (∅)
catalog	∅ <ø> (∅)
clusters_keeper	∅ <ø> (∅)
dask_sidecar	∅ <ø> (∅)
datcore_adapter	∅ <ø> (∅)
director	∅ <ø> (∅)
director_v2	77.62% <ø> (-13.40%)	⬇️
dynamic_scheduler	∅ <ø> (∅)
dynamic_sidecar	88.83% <ø> (-1.36%)	⬇️
efs_guardian	∅ <ø> (∅)
invitations	93.28% <ø> (ø)
payments	∅ <ø> (∅)
resource_usage_tracker	∅ <ø> (∅)
storage	∅ <ø> (∅)
webclient	∅ <ø> (∅)
webserver	58.66% <ø> (-27.07%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8ce837...e1e3e42. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud