Add write cloud buckets to docs #514
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #514 +/- ##
=======================================
Coverage 68.81% 68.81%
=======================================
Files 14 14
Lines 2097 2097
=======================================
Hits 1443 1443
Misses 654 654 🚀 New features to boost your workflow:
|
| savedataset(ds; path="s3://my_bucket/my_object", driver=:zarr) | ||
| ``` | ||
|
|
||
| Note that arguments `path` and `driver` can also be used to create `OutDims` in `mapCube`, enabling writing results of a computation directly to cloud object storage. |
There was a problem hiding this comment.
we are trying to the transition to xmap instead of mapCube (which will be removed soon).
There was a problem hiding this comment.
Then we just search all doc files and adjust mapCube to xmap accordingly. Needs to be done with any file.
There was a problem hiding this comment.
currently, in the compute section we did update the examples, but we also still have the old way, so that users can start the transition. Bottom line, for now we should have both ways, then in a subsequent breaking release we will remove it.
| Writing directly to S3-compatible cloud object storage is supported. | ||
| Valid credentials must be given. | ||
| Providing environmental variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` is highly recommended for username and password, respectively. | ||
| One needs to create any `AbstractAWSConfig` and activate it with `AWS.global_aws_config`, e. g. using [MinIO](https://expandingman.gitlab.io/Minio.jl/) for self-hosted storage: |
There was a problem hiding this comment.
maybe do a dummy AbstractAWSConfig object as well. Simply naming that here doesn't help much.
There was a problem hiding this comment.
MinioConfig here is the AbstractAWSConfig. This is why I added the example code underneath.
There was a problem hiding this comment.
oh, my bad, I was thinking more about AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, new users usually don't know where to set those 😓 (even I don't know where at times).
2 participants
This PR aims to clarify how to use cloud object storage in
savedatasetandmapCube. Otherwise, documentation would be spread across multiple other Julia packages, e.g., AWS, AWSS3, and Minio.