AWS batch ListObjectsV2 operation: Access Denied

[harshaldeokarutd]

I am getting below error while running nextflow script on AWS batch.

_**_Command wrapper:
nxf-scratch-dir ip-172-31-79-245.ec2.internal:/tmp/nxf.nFdsuC7gIG

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
main: line 261: 161 Killed /home/ec2-user/miniconda/bin/aws --region us-east-1 s3 cp --only-show-errors "$source" "$target"
main: line 261: 162 Killed /home/ec2-user/miniconda/bin/aws --region us-east-1 s3 cp --only-show-errors "$source" "$target"
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Work dir:
s3://harshal-redshift-training/temp/0b/138a8be340f0f91c542156798498c5

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run**_

Also the custom AMI has the aws cli installed and the ec2 instance role attached to batch compute environment has the administor access and s3 full access. Can anyone please help what exactly I am missing here.

[HarryMWinters]

I'm not sure this is your issue but I solved something similar by copying the public S3 files I needed to my own private bucket and then specifying the copied files while running the pipeline.

The problem was that Nextflow was using the attached IAM role to access S3 and the public bucket I was pulling from only allowed S3 access when --no-sign-request was specified. Since I couldn't enable --no-sign-request on all S3 requests due to utilizing private buckets I copied the public data to a bucket I controlled and moved on to the next error.