Skip to content

Comments

Vendor upstream URL parsing to simplify internal config#209

Merged
kylebarron merged 3 commits intomainfrom
kyle/vendor-url-parsing
Feb 4, 2025
Merged

Vendor upstream URL parsing to simplify internal config#209
kylebarron merged 3 commits intomainfrom
kyle/vendor-url-parsing

Conversation

@kylebarron
Copy link
Member

@kylebarron kylebarron commented Feb 4, 2025

Because url has deferred parsing in the object_store builders, we need to special-case url handling. Naturally, passing url to a store with with_url means that object_store doesn't actually parse the URL until the build() method, and at that point we can no longer access config information from the built AmazonS3. Without special-casing this URL handling, pickling would fail for instances created from from_url.

Therefore, we handle this by vendoring the small amount of URL parsing from upstream. So we apply the URL parsing onto our config HashMaps and then apply those to the builder. So our configs and those used by the raw stores stay in sync.

We originally stored the url separately, but that meant that the bucket name was stored in up to three different places (bucket variable, in the url, and in the config). This simplifies the internal config by ensuring we only store each piece of config data in only one place.

@kylebarron kylebarron changed the title Vendor URL parsing to simplify internal config Vendor upstream URL parsing to simplify internal config Feb 4, 2025
@kylebarron kylebarron merged commit 67e60bf into main Feb 4, 2025
4 checks passed
@kylebarron kylebarron deleted the kyle/vendor-url-parsing branch February 4, 2025 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant