Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 73 additions & 4 deletions docs/Cluster_basics/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
- [Table of Contents](#table-of-contents)
- [What is a S3 Storage?](#what-is-a-s3-storage)
- [How to access](#how-to-access)
- [Terminal](#terminal)
- [Terminal from IRB cluster](#terminal-from-irb-cluster)
- [Open OnDemand](#open-ondemand)
- [Mount bucket on local](#mount-bucket-on-local)
- [How to use it](#how-to-use-it)
- [Seqera and Nextflow](#seqera-and-nextflow)
- [Python](#python)
Expand Down Expand Up @@ -54,7 +56,7 @@ From now on, access to S3 is only possible via the terminal (e.g., to navigate,
If you only want to use S3 files in your Nextflow pipelines through [seqera platform](https://cloud.seqera.io/), no
further setup is needed. Credentials are already configured in the pipelines; you only need to export them.

### Terminal
### Terminal from IRB cluster

To use it via terminal, once in the IRB cluster:

Expand Down Expand Up @@ -130,7 +132,74 @@ spack load stu
stu
```

![alt text](https://github.com/lusingander/stu/raw/master/img/demo.gif)
![stu TUI demo](https://github.com/lusingander/stu/raw/master/img/demo.gif)

### Open OnDemand

In addition to command-line from the IRB cluster, you can also
access S3 via Open OnDemand. The Open OnDemand S3 interface is read-only: you can browse and
download files, but you cannot modify existing objects or upload new ones from this interface.

1. Go to
[Open OnDemand Dashboard](https://ood.sc.irbbarcelona.org/pun/sys/dashboard/)
and log in with your LDAP credentials.
2. Click on "Home Directory System installed App".

![Open OnDemand Home Directory System app screenshot](../assets/images/ooo-home-directory.png)

3. Select the S3 storage on the left panel and navigate through the folders.
You can read PDFs and text files directly in the browser.
To view Excel or HTML files, you need to download them first.

![Open OnDemand S3 storage browser screenshot](../assets/images/ooo-s3.png)

### Mount bucket on local

It is also possible to mount the S3 storage as a local partition (as we do with MacGyver and the workspace/).
This is a read-only option.

Below are the instructions for Ubuntu. It is also possible on Windows and Mac; see the official IRB documentation:
[IRB documentation.](https://irb.app.lumapps.com/home/ls/content/5695990366740739/archive-to-s3-minio)

1. Create a conda environment with rclone

```bash
conda create -n s3 rclone -c conda-forge -y
conda activate s3
```

2. Download and run the app from IT

```bash
wget https://github.com/its-irb/irb-storage-public-scripts/releases/latest/download/minio-rclone-copy-GUI-linux
chmod +x minio-rclone-copy-GUI-linux
./minio-rclone-copy-GUI-linux
```

3. Mount:

Login using your LDAP credentials.

Select the s3:

![S3 selection screen](../assets/images/s3-1.png)

If your Minio credentials are not available or expired, it will offer to renew them.

![Minio credentials renewal prompt](../assets/images/s3-2.png)

In the destination path, put the s3 bucket name and click "Mount destination folder":

![Mount destination folder dialog](../assets/images/s3-3.png)

And that's it. Now you can access from the file browser and the terminal:

![S3 mounted in file browser](../assets/images/s3-4.png)

![S3 mounted in terminal](../assets/images/s3-5.png)

This tool can be used also for moving data between the netapp and the s3,
**but this functionality is still in a testing phase and should be used with caution**.

## How to use it

Expand All @@ -139,7 +208,7 @@ stu
As mentioned, you don’t need to include any credentials explicitly in Seqera to run a job. When accessing S3 data
from a pipeline, remember to select the secrets and include the info in the config:

![alt text](../assets/images/s3_secrets.png)
![S3 secrets configuration screenshot](../assets/images/s3_secrets.png)

If using nextflow in a terminal, you can add the credentials to the *nextflow.config*

Expand Down
Binary file added docs/assets/images/ooo-home-directory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/ooo-s3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.