Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 65 additions & 2 deletions docs/Cluster_basics/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
- [Table of Contents](#table-of-contents)
- [What is a S3 Storage?](#what-is-a-s3-storage)
- [How to access](#how-to-access)
- [Terminal](#terminal)
- [Terminal from IRB cluster](#terminal-from-irb-cluster)
- [Open onDemand](#open-ondemand)
- [Mount bucket on local](#mount-bucket-on-local)
- [How to use it](#how-to-use-it)
- [Seqera and Nextflow](#seqera-and-nextflow)
- [Python](#python)
Expand Down Expand Up @@ -54,7 +56,7 @@
If you only want to use S3 files in your Nextflow pipelines through [seqera platform](https://cloud.seqera.io/), no
further setup is needed. Credentials are already configured in the pipelines; you only need to export them.

### Terminal
### Terminal from IRB cluster

To use it via terminal, once in the IRB cluster:

Expand Down Expand Up @@ -132,6 +134,67 @@

![alt text](https://github.com/lusingander/stu/raw/master/img/demo.gif)

### Open OnDemand

Read-Only Mode — You can browse and download files but cannot modify or upload from this interface.

1. Go to https://ood.sc.irbbarcelona.org/pun/sys/dashboard/ and log in with your LDAP credentials.

2. Click on "Home Directory System installed App".

![alt text](../assets/images/ooo-home-directory.png)

3. Select the S3 storage on the left panel and navigate through the folders.

Check failure on line 147 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:147:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 3; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md

Check failure on line 147 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:147:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 3; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md
You can read PDFs and text files directly in the browser.
To view Excel or HTML files, you need to download them first.

![alt text](../assets/images/ooo-s3.png)

### Mount bucket on local

It is also possible to mount the s3 storage as a local partition (as we do with macgyver and
the workspace/).
Here are the instructions for ubuntu, but it is also possible using windows and mac, check
the official IRB documentation.
[IRB documentation.](https://irb.app.lumapps.com/home/ls/content/5695990366740739/archive-to-s3-minio)

1. Create a conda enviroment with rclone.
```
conda create -n s3 rclone -c conda-forge -y

Check failure on line 163 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Fenced code blocks should have a language specified

docs/Cluster_basics/s3.md:163 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md

Check failure on line 163 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Fenced code blocks should have a language specified

docs/Cluster_basics/s3.md:163 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md
conda activate s3
```

2. Download and run the app from IT
```

Check failure on line 168 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:168:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 2; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md

Check failure on line 168 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:168:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 2; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md
wget https://github.com/its-irb/irb-storage-public-scripts/releases/latest/download/minio-rclone-copy-GUI-linux
chmod minio-rclone-copy-GUI-linux

Check failure on line 170 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Fenced code blocks should have a language specified

docs/Cluster_basics/s3.md:170 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md

Check failure on line 170 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Fenced code blocks should have a language specified

docs/Cluster_basics/s3.md:170 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md
./minio-rclone-copy-GUI-linux
```

3. Mount:

Login using your LDAP credentials.

Check failure on line 176 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:176:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 3; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md

Check failure on line 176 in docs/Cluster_basics/s3.md

View workflow job for this annotation

GitHub Actions / Lint Markdown Files

Ordered list item prefix

docs/Cluster_basics/s3.md:176:1 MD029/ol-prefix Ordered list item prefix [Expected: 1; Actual: 3; Style: 1/1/1] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md029.md

Select the s3:

![alt text](../assets/images/s3-1.png)

If your Minio credentials are not available or expired, it will offer to renew them.

![alt text](../assets/images/s3-2.png)

In the estination path, put the s3 bucket name and click "Mount destination folder":

![alt text](../assets/images/s3-3.png)

And that's it. Now you can access from the file browser and the terminal:

![alt text](../assets/images/s3-4.png)

![alt text](../assets/images/s3-5.png)

This tool can be used also for moving data between the netapp and the s3, **but it still on testing phase**.

## How to use it

### Seqera and Nextflow
Expand Down
Binary file added docs/assets/images/ooo-home-directory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/ooo-s3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/images/s3-5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading