-
Notifications
You must be signed in to change notification settings - Fork 75
Description
Title
Improve import time by lazy-loading cloud provider modules
Description
Problem
Importing cloudpathlib currently takes ~435ms because it eagerly loads all cloud provider SDKs (google-cloud-storage, boto3, azure-storage-blob) at import time, even if the user only needs one provider.
This is problematic for:
- CLI tools where startup latency matters
- Serverless functions where cold start time is critical
- Applications that only use one cloud provider but pay the import cost for all three
Benchmarks
Measured with Python 3.12 (5 runs, mean ± std):
| Scenario | Current | With Lazy Loading | Improvement |
|---|---|---|---|
import cloudpathlib |
435ms ± 24ms | 22ms ± 1ms | 95% faster |
from cloudpathlib import S3Path |
410ms ± 3ms | 23ms ± 0ms | 94% faster |
from cloudpathlib import GSPath |
410ms ± 3ms | 24ms ± 1ms | 94% faster |
from cloudpathlib import AzureBlobPath |
413ms ± 7ms | 41ms ± 1ms | 90% faster |
Additionally, the current implementation loads all three SDKs even when importing just one path class. With lazy loading, only the SDK for the provider being used gets loaded.
Proposed Solution
Implement lazy loading via __getattr__ in the __init__.py files:
cloudpathlib/__init__.pycloudpathlib/s3/__init__.pycloudpathlib/gs/__init__.pycloudpathlib/azure/__init__.py
This defers loading cloud SDKs until they are actually accessed, while preserving static type hints via TYPE_CHECKING blocks.
Example
Before:
import cloudpathlib # Takes ~435ms, loads all SDKsAfter:
import cloudpathlib # Takes ~22ms, no SDKs loaded
from cloudpathlib import S3Path # Only loads boto3 when S3Path is accessed
Implementation Notes
- Use
__getattr__withTYPE_CHECKINGblocks for static type hints (IDE support preserved) - Fix absolute import in
cloudpath.py(from cloudpathlib.enums→from .enums) - Move
anypathimport to function-local incloudpath.pyto avoid circular imports - Add tests to verify lazy loading behavior and prevent regressions
I have a working implementation ready to submit as a PR.