Skip to content

Commit 8298f02

Browse files
committed
Update docs & home to ver 20.07.1
1 parent b01fec9 commit 8298f02

File tree

156 files changed

+57072
-10579
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

156 files changed

+57072
-10579
lines changed
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
.. _amazons3-page:
2+
3+
*******************
4+
Amazon S3 storage
5+
*******************
6+
7+
Nextflow includes the support for Amazon S3 storage. Files stored in a S3 bucket can be accessed
8+
transparently in your pipeline script like any other file in the local file system.
9+
10+
S3 path
11+
---------
12+
In order to access a S3 file you only need to prefix the file path with the ``s3`` schema and the `bucket` name
13+
where it is stored.
14+
15+
For example if you need to access the file ``/data/sequences.fa`` stored in a bucket with name ``my-bucket``,
16+
that file can be accessed using the following fully qualified path::
17+
18+
s3://my-bucket/data/sequences.fa
19+
20+
21+
The usual file operations can be applied on a path handle created using the above notation. For example the content
22+
of a S3 file can be printed as shown below::
23+
24+
println file('s3://my-bucket/data/sequences.fa').text
25+
26+
27+
See section :ref:`script-file-io` to learn more about available file operations.
28+
29+
30+
31+
32+
Security credentials
33+
---------------------
34+
35+
Amazon access credentials can be provided in two ways:
36+
37+
#. Using AWS access and secret keys in your pipeline configuration.
38+
#. Using IAM roles to grant access to S3 storage on Amazon EC2 instances.
39+
40+
AWS access and secret keys
41+
===========================
42+
43+
The AWS access and secret keys can be specified by using the ``aws`` section in the ``nextflow.config`` configuration.
44+
file as shown below::
45+
46+
aws {
47+
accessKey = '<Your AWS access key>'
48+
secretKey = '<Your AWS secret key>'
49+
region = '<AWS region identifier>'
50+
}
51+
52+
53+
If the access credentials are not found in the above file, Nextflow looks for AWS credentials in a number of different
54+
places, including environment variables and local AWS configuration files.
55+
56+
57+
Nextflow looks for AWS credentials in the following order:
58+
59+
#. the ``nextflow.config`` file in the pipeline execution directory
60+
#. the environment variables ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY``
61+
#. the environment variables ``AWS_ACCESS_KEY`` and ``AWS_SECRET_KEY``
62+
#. the `default` profile in the AWS credentials file located at ``~/.aws/credentials``
63+
#. the `default` profile in the AWS client configuration file located at ``~/.aws/config``
64+
#. the temporary AWS credentials provided by an IAM instance role. See `IAM Roles <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html>`_ documentation for details.
65+
66+
67+
More information regarding `AWS Security Credentials <http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html>`_
68+
are available in Amazon documentation.
69+
70+
IAM roles Amazon EC2 instances
71+
================================
72+
73+
When running your pipeline into a Ec2 instance, IAM roles can be used to grant access to AWS resources.
74+
75+
In this scenario, you only need to launch the Ec2 instance specifying a IAM role which includes a
76+
`S3 full access` policy. Nextflow will detected and acquire automatically the access grant to the S3 storage,
77+
without any further configuration.
78+
79+
Learn more about `Using IAM Roles to Delegate Permissions to Applications that Run on Amazon EC2 <http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-usingrole-ec2instance.html>`_ on Amazon
80+
documentation.
81+
82+
Advanced configuration
83+
-----------------------
84+
85+
Read :ref:`AWS configuration<config-aws>` section to learn more about advanced S3 client configuration options.
86+
87+
88+
89+
90+
91+
92+

0 commit comments

Comments
 (0)