Skip to content

Add write cloud buckets to docs #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lazarusA merged 1 commit into from
Jul 15, 2025
Merged

Add write cloud buckets to docs #514

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/src/UserGuide/write.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,25 @@ nothing # hide

More on [Zarr Compressors](https://juliaio.github.io/Zarr.jl/latest/reference/#Compressors). Also, if you use this option and don't notice a significant improvement, please feel free to open an issue or start a discussion.

## Write to cloud buckets

Writing directly to S3-compatible cloud object storage is supported.
Valid credentials must be given.
Providing environmental variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` is highly recommended for username and password, respectively.
One needs to create any `AbstractAWSConfig` and activate it with `AWS.global_aws_config`, e. g. using [MinIO](https://expandingman.gitlab.io/Minio.jl/) for self-hosted storage:
Copy link
Collaborator

@lazarusA lazarusA Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe do a dummy AbstractAWSConfig object as well. Simply naming that here doesn't help much.

Copy link
Member Author

@danlooo danlooo Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MinioConfig here is the AbstractAWSConfig. This is why I added the example code underneath.

Copy link
Collaborator

@lazarusA lazarusA Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, my bad, I was thinking more about AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, new users usually don't know where to set those 😓 (even I don't know where at times).


```julia
using AWS
using Minio

# assume env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are available
minio_config = MinioConfig("https://s3.example.com:9000")
AWS.global_aws_config(minio_config)
savedataset(ds; path="s3://my_bucket/my_object", driver=:zarr)
```

Note that arguments `path` and `driver` can also be used to create `OutDims` in `mapCube`, enabling writing results of a computation directly to cloud object storage.
Copy link
Collaborator

@lazarusA lazarusA Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are trying to the transition to xmap instead of mapCube (which will be removed soon).

Copy link
Member Author

@danlooo danlooo Jul 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we just search all doc files and adjust mapCube to xmap accordingly. Needs to be done with any file.

Copy link
Collaborator

@lazarusA lazarusA Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently, in the compute section we did update the examples, but we also still have the old way, so that users can start the transition. Bottom line, for now we should have both ways, then in a subsequent breaking release we will remove it.


## Write NetCDF

Save a single YAXArray to a directory:
Expand Down
Loading