Store files in cloud object storage #21
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillywoodfield
force-pushed
the
18-remove-deleted-datasets
branch
from
318b683 to
66c7867
Compare
tillywoodfield
force-pushed
the
8-storage-bucket
branch
from
c817cb7 to
bf71263
Compare
jarofgreen
reviewed
Feb 11, 2025
| session = boto3.session.Session() | ||
| return session.client( | ||
| "s3", | ||
| endpoint_url=f"https://{BUCKET_REGION}.digitaloceanspaces.com/", |
There was a problem hiding this comment.
The whole Endpoint URL could be a env var? Allows changing provider easily.
There was a problem hiding this comment.
Ah, but then maybe the addressing_style config below should be a config? Maybe then it gets confusing.
There was a problem hiding this comment.
Yeah, I also ran into an issue later down the line where the boto3 client doesn't return the public URL of the uploaded file, so we have to construct that ourselves, which can be different depending on the provider, and it's easier to have all the parts, rather than a single URL. This ended up being the nicest way I could find
jarofgreen
approved these changes
Feb 11, 2025
tillywoodfield
force-pushed
the
8-storage-bucket
branch
from
bf71263 to
59b8da6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #20
Closes #8