Skip to content

Conversation

@GitHK
Copy link
Contributor

@GitHK GitHK commented Aug 14, 2025

What do these changes do?

Fix issue with permissions inside agent. The same issues was present before inside the dynamic-sidecar. Applying the same fix as before. Moved some command code to servicelib.

Unhandled exception:
Traceback (most recent call last):
  File "/home/scu/.venv/lib/python3.11/site-packages/servicelib/logging_utils.py", line 570, in log_catch
    yield
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_agent/services/docker_utils.py", line 101, in remove_volume
    await _backup_volume(app, docker, volume_name=volume_name)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_agent/services/docker_utils.py", line 88, in _backup_volume
    await backup_volume(app, volume_details, volume_name)
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_agent/services/backup.py", line 199, in backup_volume
    await _store_in_s3(
  File "/home/scu/.venv/lib/python3.11/site-packages/simcore_service_agent/services/backup.py", line 192, in _store_in_s3
    raise RuntimeError(msg)
RuntimeError: Shell subprocesses yielded nonzero error code 1 for command rclone --config /tmp/rclone_config_8e520042-2ec1-491c-aab6-14c79500d81a.ini --low-level-retries 3 --retries 3 --transfers 5 --buffer-size 0M --stats 5s --stats-one-line sync /docker/volumes/dyv_1741776911_6746a01f-cf9d-4c5e-85f1-5c93c4acec09_e74a2f91-c395-4a38-b614-aeb56e5baa99_ecapskrow_krow_ums_emoh_/_data dst:osparc-simcore-zombie-dy-volume-saves/production-simcore/76763b80-f291-11ef-b262-0242ac170593/e74a2f91-c395-4a38-b614-aeb56e5baa99/1741776911_6746a01f-cf9d-4c5e-85f1-5c93c4acec09/home_smu_work_workspace --verbose --exclude .hidden_do_not_remove --exclude key_values.json
2025/08/14 08:26:20 ERROR : .Trash-8005: failed to open directory ".Trash-8005": open /docker/volumes/dyv_1741776911_6746a01f-cf9d-4c5e-85f1-5c93c4acec09_e74a2f91-c395-4a38-b614-aeb56e5baa99_ecapskrow_krow_ums_emoh_/_data/.Trash-8005: permission denied

Related issue/s

How to test

Dev-ops

@GitHK GitHK self-assigned this Aug 14, 2025
@GitHK GitHK added the a:agent agent service label Aug 14, 2025
@GitHK GitHK added this to the Voyager milestone Aug 14, 2025
@codecov
Copy link

codecov bot commented Aug 14, 2025

Codecov Report

❌ Patch coverage is 96.55172% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 88.01%. Comparing base (cbf29d6) to head (e927433).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8214      +/-   ##
==========================================
- Coverage   88.08%   88.01%   -0.08%     
==========================================
  Files        1910     1487     -423     
  Lines       73501    61112   -12389     
  Branches     1296      650     -646     
==========================================
- Hits        64744    53787   -10957     
+ Misses       8368     7094    -1274     
+ Partials      389      231     -158     
Flag Coverage Δ
integrationtests 64.21% <100.00%> (-0.02%) ⬇️
unittests 86.37% <96.55%> (-0.35%) ⬇️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library 71.86% <92.30%> (+0.16%) ⬆️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.03% <ø> (ø)
agent 93.90% <100.00%> (+0.08%) ⬆️
api_server 93.26% <ø> (ø)
autoscaling 95.89% <ø> (ø)
catalog 92.34% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 92.15% <ø> (+0.56%) ⬆️
datcore_adapter 97.94% <ø> (ø)
director 76.23% <ø> (+0.09%) ⬆️
director_v2 90.91% <ø> (-0.02%) ⬇️
dynamic_scheduler 96.27% <ø> (ø)
dynamic_sidecar 90.05% <100.00%> (-0.03%) ⬇️
efs_guardian 89.62% <ø> (ø)
invitations 91.44% <ø> (ø)
payments 92.61% <ø> (ø)
resource_usage_tracker 92.18% <ø> (ø)
storage 86.63% <ø> (+0.29%) ⬆️
webclient ∅ <ø> (∅)
webserver 88.29% <ø> (-0.03%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cbf29d6...e927433. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@GitHK GitHK added a:services-library issues on packages/service-libs a:dynamic-sidecar dynamic-sidecar service labels Aug 14, 2025
@GitHK GitHK added the t:maintenance Some planned maintenance work label Aug 14, 2025
@mergify
Copy link
Contributor

mergify bot commented Aug 14, 2025

🧪 CI Insights

Here's what we observed from your CI run for e927433.

✅ Passed Jobs With Interesting Signals

Pipeline Job Signal Health on base branch Retries 🔍 CI Insights 📄 Logs
CI [unit] storage (3.11, ubuntu-24.04) Base branch is broken, but the job passed. Looks like this might be a real fix 💪 Broken 0 View View
unit-tests Base branch is broken, but the job passed. Looks like this might be a real fix 💪 Broken 0 View View

@GitHK GitHK marked this pull request as ready for review August 14, 2025 09:15
@GitHK GitHK requested a review from pcrespov as a code owner August 14, 2025 09:15
Copy link
Collaborator

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a permission issue with volume backups in the agent service by applying the same solution previously used in the dynamic-sidecar. The fix ensures proper file permissions before backing up volumes to prevent permission denied errors during rclone operations.

Key changes:

  • Moved container utility functions from dynamic-sidecar to shared servicelib package
  • Added permission fix to agent backup process using chmod command
  • Updated imports across services to use the centralized container utilities

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
services/agent/src/simcore_service_agent/services/backup.py Added permission fix function and logic to ensure readable permissions before backup
packages/service-library/src/servicelib/container_utils.py Moved container execution utilities and error classes from dynamic-sidecar to shared library
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/core/errors.py Removed container execution error classes that were moved to servicelib
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/prometheus_metrics.py Updated imports to use servicelib container utilities
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/long_running_tasks_utils.py Updated imports to use servicelib container utilities
services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/api/rest/containers.py Updated imports to use servicelib container utilities
services/agent/tests/unit/test_services_backup.py Added mock container fixture for testing backup functionality
packages/service-library/tests/test_container_utils.py Updated test imports to use servicelib container utilities

Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot

Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@sonarqubecloud
Copy link

@sanderegg sanderegg merged commit 068394c into ITISFoundation:master Aug 18, 2025
95 checks passed
@GitHK GitHK deleted the pr-osparc-fix-agent-permission-issue branch August 18, 2025 08:58
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Sep 2, 2025
61 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a:agent agent service a:dynamic-sidecar dynamic-sidecar service a:services-library issues on packages/service-libs t:maintenance Some planned maintenance work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants