|
| 1 | +.. _amazons3-page: |
| 2 | + |
| 3 | +******************* |
| 4 | +Amazon S3 storage |
| 5 | +******************* |
| 6 | + |
| 7 | +Nextflow includes the support for Amazon S3 storage. Files stored in a S3 bucket can be accessed |
| 8 | +transparently in your pipeline script like any other file in the local file system. |
| 9 | + |
| 10 | +S3 path |
| 11 | +--------- |
| 12 | +In order to access a S3 file you only need to prefix the file path with the ``s3`` schema and the `bucket` name |
| 13 | +where it is stored. |
| 14 | + |
| 15 | +For example if you need to access the file ``/data/sequences.fa`` stored in a bucket with name ``my-bucket``, |
| 16 | +that file can be accessed using the following fully qualified path:: |
| 17 | + |
| 18 | + s3://my-bucket/data/sequences.fa |
| 19 | + |
| 20 | + |
| 21 | +The usual file operations can be applied on a path handle created using the above notation. For example the content |
| 22 | +of a S3 file can be printed as shown below:: |
| 23 | + |
| 24 | + println file('s3://my-bucket/data/sequences.fa').text |
| 25 | + |
| 26 | + |
| 27 | +See section :ref:`script-file-io` to learn more about available file operations. |
| 28 | + |
| 29 | + |
| 30 | + |
| 31 | + |
| 32 | +Security credentials |
| 33 | +--------------------- |
| 34 | + |
| 35 | +Amazon access credentials can be provided in two ways: |
| 36 | + |
| 37 | +#. Using AWS access and secret keys in your pipeline configuration. |
| 38 | +#. Using IAM roles to grant access to S3 storage on Amazon EC2 instances. |
| 39 | + |
| 40 | +AWS access and secret keys |
| 41 | +=========================== |
| 42 | + |
| 43 | +The AWS access and secret keys can be specified by using the ``aws`` section in the ``nextflow.config`` configuration. |
| 44 | +file as shown below:: |
| 45 | + |
| 46 | + aws { |
| 47 | + accessKey = '<Your AWS access key>' |
| 48 | + secretKey = '<Your AWS secret key>' |
| 49 | + region = '<AWS region identifier>' |
| 50 | + } |
| 51 | + |
| 52 | + |
| 53 | +If the access credentials are not found in the above file, Nextflow looks for AWS credentials in a number of different |
| 54 | +places, including environment variables and local AWS configuration files. |
| 55 | + |
| 56 | + |
| 57 | +Nextflow looks for AWS credentials in the following order: |
| 58 | + |
| 59 | + #. the ``nextflow.config`` file in the pipeline execution directory |
| 60 | + #. the environment variables ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` |
| 61 | + #. the environment variables ``AWS_ACCESS_KEY`` and ``AWS_SECRET_KEY`` |
| 62 | + #. the `default` profile in the AWS credentials file located at ``~/.aws/credentials`` |
| 63 | + #. the `default` profile in the AWS client configuration file located at ``~/.aws/config`` |
| 64 | + #. the temporary AWS credentials provided by an IAM instance role. See `IAM Roles <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html>`_ documentation for details. |
| 65 | + |
| 66 | + |
| 67 | +More information regarding `AWS Security Credentials <http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html>`_ |
| 68 | +are available in Amazon documentation. |
| 69 | + |
| 70 | +IAM roles Amazon EC2 instances |
| 71 | +================================ |
| 72 | + |
| 73 | +When running your pipeline into a Ec2 instance, IAM roles can be used to grant access to AWS resources. |
| 74 | + |
| 75 | +In this scenario, you only need to launch the Ec2 instance specifying a IAM role which includes a |
| 76 | +`S3 full access` policy. Nextflow will detected and acquire automatically the access grant to the S3 storage, |
| 77 | +without any further configuration. |
| 78 | + |
| 79 | +Learn more about `Using IAM Roles to Delegate Permissions to Applications that Run on Amazon EC2 <http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-usingrole-ec2instance.html>`_ on Amazon |
| 80 | +documentation. |
| 81 | + |
| 82 | +Advanced configuration |
| 83 | +----------------------- |
| 84 | + |
| 85 | +Read :ref:`AWS configuration<config-aws>` section to learn more about advanced S3 client configuration options. |
| 86 | + |
| 87 | + |
| 88 | + |
| 89 | + |
| 90 | + |
| 91 | + |
| 92 | + |
0 commit comments