Using a non-AWS S3 filesystem (Ceph) with custom endpoint

[davidparks21]

I'm getting the error:

ERROR ~ The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 3C37RNVAQ69WZJ1X; S3 Extended Request ID: jTZ8NlI7kQ5/4uEQraxFv2f0Uoe7rxD68lRD3lCHmytwFBZnauBE9B+OVF5K1eGTGod0Un6nw16az0T7w7wj2A==; Proxy: null)

This error is common if the endpoint isn't set, which is what I suspect is the case here. We're using a non AWS S3 filesystem (Ceph on our k8s cluster).

Here's my nextflow.config (this is a new setup, just a proof of concept test setup).

process {
    executor = 'k8s'
}

k8s {
    namespace = 'braingeneers'
    computeResourceType = 'Job'
}

aws {
    endpoint = 's3-west.nrp-nautilus.io'
    pathStyleAccess = true
}

wave {
    enabled = true
}

fusion {
    enabled = true
    exportAwsAccessKeys = true
}

workDir = 's3://braingeneersdev/nextflow/'

I can confirm inside the docker container that I can access our bucket as expected.

docker run --rm -it -v ~/.aws/credentials:/root/.aws/credentials nextflow/nextflow:latest bash
yum install awscli
aws --endpoint https://s3-west.nrp-nautilus.io s3 ls s3://braingeneersdev/
2021-03-25 00:19:16         12 test_file.txt

We use the nextflow docker container and volume mount the ~/.aws/credentials file used in the aws example above. I didn't set the secret and key because the credentials file is mounted to the default location in the docker container, as shown in the example above.

The docs seem to suggest this configuration is valid:

• https://nextflow.io/docs/latest/config.html#scope-aws
• https://nextflow.io/docs/latest/fusion.html

[pditommaso]

Please include the full .nextflow.log file

[davidparks21]

bash-4.2# cat .nextflow.log
Aug-10 16:13:03.277 [main] DEBUG nextflow.cli.Launcher - $> nextflow run /workflows/demo_workflow.nf --xyz 1970-01-01-e-demo
Aug-10 16:13:03.326 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.08.0-edge
Aug-10 16:13:03.344 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Aug-10 16:13:03.352 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Aug-10 16:13:03.353 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Aug-10 16:13:03.355 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Aug-10 16:13:03.362 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Aug-10 16:13:03.378 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /workflows/nextflow.config
Aug-10 16:13:03.380 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /workflows/nextflow.config
Aug-10 16:13:03.403 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Aug-10 16:13:03.893 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Aug-10 16:13:03.907 [main] INFO  nextflow.cli.CmdRun - Launching `/workflows/demo_workflow.nf` [agitated_edison] DSL2 - revision: c31f00e55f
Aug-10 16:13:03.908 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[[email protected]]
Aug-10 16:13:03.909 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected], [email protected]]
Aug-10 16:13:03.909 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.1.1
Aug-10 16:13:03.911 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin [email protected]
Aug-10 16:13:06.828 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-amazon-2.1.1.zip' in 'nf-amazon-2.1.1'
Aug-10 16:13:06.836 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Aug-10 16:13:06.836 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Aug-10 16:13:06.851 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Aug-10 16:13:06.865 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Aug-10 16:13:06.866 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-wave version: 0.11.1
Aug-10 16:13:06.866 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin [email protected]
Aug-10 16:13:07.715 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-wave-0.11.1.zip' in 'nf-wave-0.11.1'
Aug-10 16:13:07.716 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Aug-10 16:13:07.716 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Aug-10 16:13:07.722 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Aug-10 16:13:07.733 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /.nextflow/secrets/store.json
Aug-10 16:13:07.737 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@4aa2877c] - activable => nextflow.secret.LocalSecretsProvider@4aa2877c
Aug-10 16:13:07.798 [main] DEBUG nextflow.Session - Session UUID: e8c1747e-dffe-400d-8f12-8c05a798260a
Aug-10 16:13:07.798 [main] DEBUG nextflow.Session - Run name: agitated_edison
Aug-10 16:13:07.799 [main] DEBUG nextflow.Session - Executor pool size: 16
Aug-10 16:13:07.811 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Aug-10 16:13:07.822 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {max_error_retry=5}
Aug-10 16:13:08.064 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Using AWS credentials file '/root/.aws/credentials'
Aug-10 16:13:08.263 [main] DEBUG nextflow.cloud.aws.nio.S3Client - Setting S3 glacierRetrievalTier=null
Aug-10 16:13:08.286 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Aug-10 16:13:08.293 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Aug-10 16:13:08.303 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 23.08.0-edge build 5872
  Created: 05-08-2023 14:34 UTC 
  System: Linux 6.2.6-76060206-generic
  Runtime: Groovy 3.0.18 on OpenJDK 64-Bit Server VM 17.0.8+7-LTS
  Encoding: UTF-8 (UTF-8)
  Process: 7@8d4c6d855042 [172.17.0.3]
  CPUs: 16 - Mem: 62.5 GB (4.6 GB) - Swap: 80 GB (78.4 GB)
Aug-10 16:13:08.914 [main] ERROR nextflow.cli.Launcher - @unknown
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 1BPDXF70J0A111G5; S3 Extended Request ID: nioVlwk4Go0hp9j801kJGt2yPmxPGP12+TsRNPPmH3+JO6OLNK8R2rRMKXho2ebNfr8EUou2uHg=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
        at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
        at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6532)
        at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
        at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
        at nextflow.cloud.aws.nio.S3Client.putObject(S3Client.java:210)
        at nextflow.cloud.aws.nio.S3FileSystemProvider.createDirectory(S3FileSystemProvider.java:490)
        at java.base/java.nio.file.Files.createDirectory(Files.java:700)
        at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
        at java.base/java.nio.file.Files.createDirectories(Files.java:753)
        at java_nio_file_Files$createDirectories.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at java_nio_file_Files$createDirectories.call(Unknown Source)
        at nextflow.extension.FilesEx.mkdirs(FilesEx.groovy:493)
        at nextflow.Session.init(Session.groovy:364)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:122)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:494)
        at nextflow.cli.Launcher.main(Launcher.groovy:653)

[davidparks21]

This nextflow error looks suspiciously similar to what you get from the aws command line if you don't set the endpoint URL (it defaults to Amazon AWS S3 endpoint):

# note the lack of --endpoint below
$ aws s3 ls s3://braingeneers/

An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

[pditommaso]

Looking at the logs it says AWS S3 config properties: {max_error_retry=5}, my understanding is that you are using a custom endpoint that should be reported in that settings. Likely your config is malformed

[davidparks21]

We're talking about the file ~/.aws/config here right? Not the nextflow.config file where endpoint is defined, right?

aws {
    endpoint = 's3-west.nrp-nautilus.io'
    pathStyleAccess = true
}

I don't normally even have a ~/.aws/config file at all, it's not needed for anything else I do (awscli, python+boto3). If that line AWS S3 config properties: {max_error_retry=5} is saying that it failed to load the ~/.aws/config file I would certainly believe it.

However, when I tried including the file ~/.aws/config it made no difference, my nextflow log was identical. I confirmed that inside the container I ran nextflow in (it's the nextflow/nextflow:latest container) I see a valid ~/.aws/config file which I regenerated by hand using aws commands only:

bash-4.2# cat ~/.aws/config
[default]
s3 =
    endpoint_url = https://s3-west.nrp-nautilus.io

This was my container run step:

docker run --rm -it  \
  --volume $(pwd)/workflows/:/workflows/   
  --volume ~/.aws/credentials:/root/.aws/credentials   
  --volume ~/.aws/config:/root/.aws/config   
  --volume ~/.kube/config:/root/.kube/config   
  nextflow/nextflow:latest bash

Within that container I ran my test workflow:

bash-4.2# nextflow run /workflows/demo_workflow.nf --xyz 1970-01-01-e-demo
N E X T F L O W  ~  version 23.08.0-edge
Launching `/workflows/demo_workflow.nf` [clever_pasteur] DSL2 - revision: c31f00e55f
Downloading plugin [email protected]
Downloading plugin [email protected]
ERROR ~ The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: TP4GZ2NETWKS7T7E; S3 Extended Request ID: AtQjGN4Pgbo6CrlrKqdpuR8hcl1U1URyDzI6M7TO3dyQNfSP/tWHNUFgJ18K6wvrLLCpLWghLak=; Proxy: null)

 -- Check '.nextflow.log' file for details

I'm not sure what dependencies nextflow might have on it, and I don't know how to read AWS S3 config properties: {max_error_retry=5} as an error. If it can't read the config file but it's expecting to, I'm confused because I've now confirmed ~/.aws/config in the default location.

I would have guessed that I didn't define endpoint in the right place in nextflow.config, but I don't see what I'd do differently. I'm still stuck here, not sure how to get nextflow to accept a non-default endpoint.

[pditommaso]

Nope, the problem is the nextflow config. it should be

aws {
  client {
    endpoint = 's3-west.nrp-nautilus.io'
    pathStyleAccess = true
  }
}

[davidparks21]

Ah, it's right there in the documentation, I just misread the aws scope page.
https://www.nextflow.io/docs/latest/config.html#config-aws

It gave the following example, but below it clearly states that the table where endpoint is found is under the client scope. I blew right past that line. My bad. Thanks a ton for that.

aws {
    accessKey = '<YOUR S3 ACCESS KEY>'
    secretKey = '<YOUR S3 SECRET KEY>'
    region = '<AWS REGION IDENTIFIER>'
    profile = '<AWS CONFIG PROFILE>' // optional
}

The logs look as expected now:

Aug-17 18:51:33.496 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {path_style_access=true, endpoint=s3-west.nrp-nautilus.io, max_error_retry=5}
Aug-17 18:51:33.688 [main] DEBUG nextflow.cloud.aws.AwsClientFactory - Merging AWS credentials file '/root/.aws/credentials' and config file '/root/.aws/config'

For completeness there was another error with the pathStyleAccess, it should have been:

aws {
  client {
    endpoint = 's3-west.nrp-nautilus.io'
    s3PathStyleAccess = true
    protocol = 'https'
  }
}

Once I corrected that it started working.

Thanks a ton for the help!

[pditommaso]

Enjoy