Skip to content

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Apr 25, 2025

What do these changes do?

our CEPH S3 is not 100% compatible anymore (or at least for our use) anymore with AWS S3 since botocore 1.36.0 was released this year.

Until CEPH is updated botocore version is locked which has implications on many other libraries.
this is now constrained on storage and dask-sidecar services which are the only one uploading data to S3 at this time.

The current CEPH S3 is incompatible with botocore, the AWS CLI, and any other tools based on botocore (which is almost everything)

Highlights on updated libraries (only updated libraries are included)

  • #packages before ~ 5
  • #packages after ~ 5
# name before after upgrade count packages
1 aioboto3 14.1.0 13.3.0 🔥 downgrade 1 storage⬆️
2 aiobotocore 2.21.1 2.17.0,2.16.0 🔥 downgrade 2 dask-sidecar⬆️
storage⬆️
3 boto3 1.37.1 1.35.93,1.35.81 🔥 downgrade 3 dask-sidecar🧪
storage⬆️🧪
4 botocore 1.37.1 1.35.93,1.35.81 🔥 downgrade 4 dask-sidecar⬆️🧪
storage⬆️🧪
5 s3transfer 0.11.3 0.10.4 🔥 downgrade 3 dask-sidecar🧪
storage⬆️🧪

Legend:

  • ⬆️ base dependency (only services because packages are floating)
  • 🧪 test dependency
  • 🔧 tool dependency

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added the t:maintenance Some planned maintenance work label Apr 25, 2025
@sanderegg sanderegg added this to the Pauwel Kwak milestone Apr 25, 2025
@sanderegg sanderegg self-assigned this Apr 25, 2025
@codecov
Copy link

codecov bot commented Apr 25, 2025

Codecov Report

Attention: Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.06%. Comparing base (cb79d90) to head (5d03e42).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7587      +/-   ##
==========================================
+ Coverage   87.69%   89.06%   +1.37%     
==========================================
  Files        1774     1245     -529     
  Lines       68426    52706   -15720     
  Branches     1125      218     -907     
==========================================
- Hits        60006    46945   -13061     
+ Misses       8113     5692    -2421     
+ Partials      307       69     -238     
Flag Coverage Δ
integrationtests 64.03% <ø> (-1.10%) ⬇️
unittests 88.17% <90.90%> (+1.29%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library 93.87% <90.90%> (-0.04%) ⬇️
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.66% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 91.27% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.64% <ø> (ø)
clusters_keeper 99.25% <ø> (ø)
dask_sidecar 91.29% <ø> (ø)
datcore_adapter 98.12% <ø> (ø)
director 76.80% <ø> (ø)
director_v2 91.40% <ø> (+0.01%) ⬆️
dynamic_scheduler 97.40% <ø> (ø)
dynamic_sidecar 90.15% <ø> (ø)
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.12% <ø> (-0.11%) ⬇️
storage 87.56% <ø> (-0.11%) ⬇️
webclient ∅ <ø> (∅)
webserver 85.70% <ø> (-0.36%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb79d90...5d03e42. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg force-pushed the maintenance/lock-boto3 branch from 04a5e4a to 5d03e42 Compare April 25, 2025 11:55
@sonarqubecloud
Copy link

@sanderegg sanderegg merged commit 1b6b386 into ITISFoundation:master Apr 25, 2025
78 of 80 checks passed
@sanderegg sanderegg deleted the maintenance/lock-boto3 branch April 25, 2025 12:01
sanderegg added a commit to sanderegg/osparc-simcore that referenced this pull request Apr 25, 2025
sanderegg added a commit to sanderegg/osparc-simcore that referenced this pull request Apr 25, 2025
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request May 8, 2025
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t:maintenance Some planned maintenance work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants