Skip to content

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Apr 25, 2025

What do these changes do?

This pull request includes updates to the AWS S3 client library and related dependencies, along with refactoring and cleanup of the codebase. The changes primarily focus on removing outdated compatibility checks, upgrading dependencies, and improving code readability and maintainability.

This should allow using the latest botocore library with our current CEPH S3.

Dependency Updates and Compatibility Changes:

  • Upgraded botocore to version 1.37.1 and aiobotocore to version 2.21.1, removing constraints on older versions that were required for CEPH S3 compatibility. [1] [2] [3] [4]
  • Updated s3transfer to version 0.11.3 and boto3 to version 1.37.1 across multiple services. [1] [2] [3]

Code Cleanup and Refactoring:

  • Removed the _check_botocore_version function and related warnings about botocore compatibility with CEPH S3. [1] [2] [3] [4]
  • Eliminated unused imports such as warnings, version, and botocore_version from s3/_client.py. [1] [2] [3]

Functional Enhancements:

  • Added request_checksum_calculation="when_required" to the S3 client configuration to ensure checksum calculations are performed when needed. [1] [2]
  • Improved the _copy_file function by refactoring the fsspec.open context manager for better readability.

Packaging and Dependency Management:

  • Adjusted requirements files across services to align with the updated dependencies, removing outdated constraints and ensuring compatibility. [1] [2] [3]

These changes collectively modernize the codebase, improve maintainability, and ensure compatibility with the latest versions of AWS-related libraries.

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added bug buggy, it does not work as expected t:maintenance Some planned maintenance work labels Apr 25, 2025
@sanderegg sanderegg added this to the Pauwel Kwak milestone Apr 25, 2025
@sanderegg sanderegg self-assigned this Apr 25, 2025
@sanderegg sanderegg changed the title 🐛Maintenance: Unlock botocore again 🐛Maintenance: Unlock botocore again 🚨🚨🚨🚨 Apr 25, 2025
@codecov
Copy link

codecov bot commented Apr 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.22%. Comparing base (1d581a9) to head (ab1ce8b).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7589      +/-   ##
==========================================
+ Coverage   87.70%   89.22%   +1.52%     
==========================================
  Files        1774     1245     -529     
  Lines       68437    52697   -15740     
  Branches     1126      217     -909     
==========================================
- Hits        60021    47020   -13001     
+ Misses       8108     5609    -2499     
+ Partials      308       68     -240     
Flag Coverage Δ
integrationtests 65.14% <ø> (-0.11%) ⬇️
unittests 88.18% <100.00%> (+1.29%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library 93.92% <100.00%> (+0.04%) ⬆️
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.66% <ø> (ø)
agent 96.46% <ø> (ø)
api_server 91.27% <ø> (ø)
autoscaling 96.08% <ø> (ø)
catalog 92.64% <ø> (ø)
clusters_keeper 99.25% <ø> (ø)
dask_sidecar 91.30% <100.00%> (+0.01%) ⬆️
datcore_adapter 98.12% <ø> (ø)
director 76.80% <ø> (ø)
director_v2 91.37% <ø> (-0.03%) ⬇️
dynamic_scheduler 97.40% <ø> (ø)
dynamic_sidecar 90.15% <ø> (ø)
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.23% <ø> (-0.06%) ⬇️
storage 87.60% <ø> (+0.10%) ⬆️
webclient ∅ <ø> (∅)
webserver 86.10% <ø> (+<0.01%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d581a9...ab1ce8b. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx sylvain!

@sanderegg sanderegg force-pushed the maintenance/unlock-botocore-again branch from 251a7fb to e4dafa7 Compare April 25, 2025 13:39
@sonarqubecloud
Copy link

@sanderegg sanderegg merged commit 6c3fca7 into ITISFoundation:master Apr 25, 2025
93 of 95 checks passed
@sanderegg sanderegg deleted the maintenance/unlock-botocore-again branch April 25, 2025 14:32
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request May 8, 2025
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug buggy, it does not work as expected t:maintenance Some planned maintenance work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants