Skip to content

move gcs and s3 deps to extras#504

Open
hirosassa wants to merge 1 commit intom3dev:masterfrom
hirosassa:extra-gcs-s3
Open

move gcs and s3 deps to extras#504
hirosassa wants to merge 1 commit intom3dev:masterfrom
hirosassa:extra-gcs-s3

Conversation

@hirosassa
Copy link
Copy Markdown
Collaborator

Summary

  • Move S3 (boto3) and GCS (google-auth, google-api-python-client) dependencies to optional extras, installable via pip install gokart[s3], pip install gokart[gcs], or pip install gokart[all]
  • Remove unused dependency uritemplate
  • Lazy-load S3/GCS-related imports, raising a clear ImportError with installation instructions when extras are missing

Motivation

pip install gokart previously bundled cloud storage libraries even when unused, increasing package size unnecessarily.

  • S3 extras (boto3 + transitive deps)
  • GCS extras (google-api-python-client + transitive deps)

By making these optional, users who don't need S3/GCS can reduce dependencies by up to ~109 MB.

Dependency size impact

Compared to gokart[all] (~110 MB of extras dependencies):

Install command Extras included Savings vs [all]
pip install gokart none -110 MB
pip install gokart[s3] S3 only (~24 MB) -86 MB
pip install gokart[gcs] GCS only (~86 MB) -24 MB
pip install gokart[all] both (baseline)

Breaking Changes

S3/GCS functionality is no longer available with a bare pip install gokart. Users must opt in:

pip install gokart[s3]   # S3 only
pip install gokart[gcs]  # GCS only
pip install gokart[all]  # both

Changes

  • pyproject.toml: Move boto3, google-auth, google-api-python-client from dependencies to optional-dependencies.
  • s3_config.py, gcs_config.py, object_storage.py, gcs_obj_metadata_client.py, s3_zip_client.py, gcs_zip_client.py: Convert top-level imports to lazy imports with helpful error messages.
  • test/test_extras.py: Add tests verifying behavior with and without extras installed.

Copy link
Copy Markdown
Contributor

@hiro-o918 hiro-o918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants