-
Notifications
You must be signed in to change notification settings - Fork 2
S3 section, refactor code structure for readability and maintainability #317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
bf7b7c4
a4f7deb
6ff970c
9ead348
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,7 +6,9 @@ | |
| - [Table of Contents](#table-of-contents) | ||
| - [What is a S3 Storage?](#what-is-a-s3-storage) | ||
| - [How to access](#how-to-access) | ||
| - [Terminal](#terminal) | ||
| - [Terminal from IRB cluster](#terminal-from-irb-cluster) | ||
| - [Open onDemand](#open-ondemand) | ||
| - [Mount bucket on local](#mount-bucket-on-local) | ||
| - [How to use it](#how-to-use-it) | ||
| - [Seqera and Nextflow](#seqera-and-nextflow) | ||
| - [Python](#python) | ||
|
|
@@ -54,7 +56,7 @@ | |
| If you only want to use S3 files in your Nextflow pipelines through [seqera platform](https://cloud.seqera.io/), no | ||
| further setup is needed. Credentials are already configured in the pipelines; you only need to export them. | ||
|
|
||
| ### Terminal | ||
| ### Terminal from IRB cluster | ||
|
|
||
| To use it via terminal, once in the IRB cluster: | ||
|
|
||
|
|
@@ -132,6 +134,67 @@ | |
|
|
||
|  | ||
|
|
||
| ### Open OnDemand | ||
|
|
||
| Read-Only Mode — You can browse and download files but cannot modify or upload from this interface. | ||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| 1. Go to https://ood.sc.irbbarcelona.org/pun/sys/dashboard/ and log in with your LDAP credentials. | ||
|
|
||
| 2. Click on "Home Directory System installed App". | ||
|
|
||
|  | ||
|
|
||
| 3. Select the S3 storage on the left panel and navigate through the folders. | ||
|
Check failure on line 147 in docs/Cluster_basics/s3.md
|
||
| You can read PDFs and text files directly in the browser. | ||
| To view Excel or HTML files, you need to download them first. | ||
|
|
||
|  | ||
|
|
||
| ### Mount bucket on local | ||
|
|
||
| It is also possible to mount the s3 storage as a local partition (as we do with macgyver and | ||
| the workspace/). | ||
| Here are the instructions for ubuntu, but it is also possible using windows and mac, check | ||
| the official IRB documentation. | ||
| [IRB documentation.](https://irb.app.lumapps.com/home/ls/content/5695990366740739/archive-to-s3-minio) | ||
migrau marked this conversation as resolved.
Show resolved
Hide resolved
migrau marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| 1. Create a conda enviroment with rclone. | ||
| ``` | ||
| conda create -n s3 rclone -c conda-forge -y | ||
|
Check failure on line 163 in docs/Cluster_basics/s3.md
|
||
| conda activate s3 | ||
| ``` | ||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| 2. Download and run the app from IT | ||
| ``` | ||
|
Check failure on line 168 in docs/Cluster_basics/s3.md
|
||
| wget https://github.com/its-irb/irb-storage-public-scripts/releases/latest/download/minio-rclone-copy-GUI-linux | ||
| chmod minio-rclone-copy-GUI-linux | ||
|
Check failure on line 170 in docs/Cluster_basics/s3.md
|
||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ./minio-rclone-copy-GUI-linux | ||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ``` | ||
|
|
||
migrau marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| 3. Mount: | ||
|
|
||
| Login using your LDAP credentials. | ||
|
Check failure on line 176 in docs/Cluster_basics/s3.md
|
||
|
|
||
| Select the s3: | ||
|
|
||
|  | ||
|
|
||
| If your Minio credentials are not available or expired, it will offer to renew them. | ||
|
|
||
|  | ||
|
|
||
| In the estination path, put the s3 bucket name and click "Mount destination folder": | ||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
|  | ||
|
|
||
| And that's it. Now you can access from the file browser and the terminal: | ||
|
|
||
|  | ||
|
|
||
|  | ||
|
|
||
| This tool can be used also for moving data between the netapp and the s3, **but it still on testing phase**. | ||
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
migrau marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## How to use it | ||
|
|
||
| ### Seqera and Nextflow | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.