I am trying to use S3 of LocalStack with Nextflow but I got runtime error.

[spksoft]

Bug report

I trying to use S3 from LocalStack by setting nextflow.config but when I run a workflow it got error

Expected behavior and actual behavior

It should display S3 files.

Steps to reproduce the problem

I have folder structure like this

.
├── Makefile
├── data
├── docker-compose.yml
├── localstack
├── nextflow
│   ├── data
│   ├── main.nf
│   ├── nextflow.config
│   └── work
└── scripts
    └── encode_base64.py

docker-compose.yaml

version: '3.8'

services:
  localstack:
    image: localstack/localstack:latest
    container_name: localstack
    environment:
      - SERVICES=s3
      - DEBUG=1
    healthcheck:
      test:
        - CMD
        - bash
        - -c
        - awslocal s3 ls
      interval: 5s
      timeout: 10s
      start_period: 10s
    ports:
      - "4566:4566"
      - "4571:4571"
    volumes:
      - ./localstack:/var/lib/localstack

  init:
    image: amazon/aws-cli:latest
    container_name: init
    entrypoint: ["sh", "-c", "aws --endpoint-url=http://localstack:4566 s3api create-bucket --bucket test-bucket && aws --endpoint-url=http://localstack:4566 s3api put-object --bucket test-bucket --key work/"]
    environment:
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - /tmp:/tmp
    depends_on:
      localstack:
        condition: service_healthy

  nextflow:
    image: nextflow/nextflow:latest
    container_name: nextflow
    platform: linux/amd64
    volumes:
      - ./nextflow:/workspace/nextflow
      - ./scripts:/workspace/scripts
    working_dir: /workspace/nextflow
    environment:
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_S3_ENDPOINT=http://localstack:4566
    command: bash -c "while true; do sleep 1; done"

  python:
    image: python:3.9-slim
    container_name: python
    volumes:
      - ./scripts:/workspace/scripts
      - ./data:/workspace/data
    working_dir: /workspace/scripts
    command: bash -c "while true; do sleep 1; done"

volumes:
  nextflow_data:
  localstack_data:

nextflow.config

aws {
    accessKey = 'test'
    secretKey = 'test'
    region = 'us-east-1'
    client {
        endpoint = 'http://localstack:4566'
        s3PathStyleAccess = true
    }
}

main.nf

println "AWS_ACCESS_KEY_ID: ${System.getenv('AWS_ACCESS_KEY_ID')}"
println "AWS_SECRET_ACCESS_KEY: ${System.getenv('AWS_SECRET_ACCESS_KEY')}"
println "AWS_DEFAULT_REGION: ${System.getenv('AWS_DEFAULT_REGION')}"
println "AWS_S3_ENDPOINT Endpoint: ${System.getenv('AWS_S3_ENDPOINT')}"

file('s3://test-bucket/work/').list().each { println it } 

and then I execute workflow by

$ docker compose up --build
$ docker exec -it nextflow nextflow run /workspace/nextflow/main.nf

Program output

Output:

 N E X T F L O W   ~  version 24.04.3

Launching `main.nf` [angry_poisson] DSL2 - revision: 3a7c65154b

AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_DEFAULT_REGION: us-east-1
S3 Endpoint: http://localstack:4566
ERROR ~ The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)

 -- Check script 'main.nf' at line: 6 or see '.nextflow.log' file for more details

.nextflow.log

Jul-28 18:23:53.891 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Jul-28 18:23:54.160 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.3
Jul-28 18:23:54.226 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/.nextflow/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jul-28 18:23:54.284 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-28 18:23:54.286 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-28 18:23:54.295 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jul-28 18:23:54.334 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jul-28 18:23:54.401 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /workspace/nextflow/nextflow.config
Jul-28 18:23:54.405 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /workspace/nextflow/nextflow.config
Jul-28 18:23:54.474 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /.nextflow/secrets/store.json
Jul-28 18:23:54.481 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@267bbe1a] - activable => nextflow.secret.LocalSecretsProvider@267bbe1a
Jul-28 18:23:54.511 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Jul-28 18:23:55.580 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Jul-28 18:23:55.599 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [angry_poisson] DSL2 - revision: 3a7c65154b
Jul-28 18:23:55.603 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-28 18:23:55.604 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jul-28 18:23:55.699 [main] DEBUG nextflow.Session - Session UUID: 15cf5789-31da-45a2-8cf8-f631afb484ab
Jul-28 18:23:55.700 [main] DEBUG nextflow.Session - Run name: angry_poisson
Jul-28 18:23:55.702 [main] DEBUG nextflow.Session - Executor pool size: 10
Jul-28 18:23:55.716 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jul-28 18:23:55.727 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=30; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jul-28 18:23:55.766 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 24.04.3 build 5916
  Created: 09-07-2024 19:35 UTC 
  System: Linux 6.6.32-linuxkit
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 17.0.10+7-LTS
  Encoding: UTF-8 (UTF-8)
  Process: 897@347678c70a13 [172.21.0.3]
  CPUs: 10 - Mem: 7.7 GB (4.7 GB) - Swap: 1024 MB (1024 MB)
Jul-28 18:23:55.834 [main] DEBUG nextflow.Session - Work-dir: /workspace/nextflow/work [UNKNOWN (0x6a656a63)]
Jul-28 18:23:55.836 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /workspace/nextflow/bin
Jul-28 18:23:55.847 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-28 18:23:55.862 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-28 18:23:55.960 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jul-28 18:23:56.029 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 11; maxThreads: 1000
Jul-28 18:23:56.175 [main] DEBUG nextflow.Session - Session start
Jul-28 18:23:56.557 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-28 18:23:56.593 [main] INFO  nextflow.script.BaseScript - AWS_ACCESS_KEY_ID: test
Jul-28 18:23:56.594 [main] INFO  nextflow.script.BaseScript - AWS_SECRET_ACCESS_KEY: test
Jul-28 18:23:56.595 [main] INFO  nextflow.script.BaseScript - AWS_DEFAULT_REGION: us-east-1
Jul-28 18:23:56.596 [main] INFO  nextflow.script.BaseScript - S3 Endpoint: http://localstack:4566
Jul-28 18:23:56.609 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 2.5.2
Jul-28 18:23:56.651 [main] INFO  org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Jul-28 18:23:56.652 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Jul-28 18:23:56.686 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Jul-28 18:23:56.710 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jul-28 18:23:56.712 [main] DEBUG nextflow.file.FileHelper - Started plugin 'nf-amazon' required to handle file: s3://test-bucket/work/
Jul-28 18:23:56.727 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jul-28 18:23:56.751 [main] DEBUG nextflow.cloud.aws.config.AwsConfig - AWS S3 config properties: {endpoint=http://localstack:4566, s_3_path_style_access=true, max_error_retry=5}
Jul-28 18:23:58.667 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_212421a51cfab87f: /workspace/nextflow/main.nf
Jul-28 18:23:58.669 [main] DEBUG nextflow.Session - Session aborted -- Cause: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)
Jul-28 18:23:58.703 [main] ERROR nextflow.cli.Launcher - @unknown
com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: Y9P3Q4J4W7SGW8VX; S3 Extended Request ID: YFh0i0iU1n4KNU4RmJzksaFiMLjbUsr+NlcUelo7AoynSaNUXjDjDdmRll3movcQReA+T90hud8=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:949)
	at nextflow.cloud.aws.nio.S3Client.listObjects(S3Client.java:137)
	at nextflow.cloud.aws.nio.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:89)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttr0(S3FileSystemProvider.java:702)
	at nextflow.cloud.aws.nio.S3FileSystemProvider.readAttributes(S3FileSystemProvider.java:686)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
	at java.base/java.nio.file.Files.isDirectory(Files.java:2322)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.isDirectory(FilesEx.groovy:601)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:59)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:57)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.listFiles(FilesEx.groovy:640)
	at nextflow.extension.FilesEx.listFiles(FilesEx.groovy)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.extension.FilesEx.list(FilesEx.groovy:628)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:59)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:57)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at groovy.lang.MetaClassImpl.doInvokeMethod(MetaClassImpl.java:1333)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1088)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at Script_212421a51cfab87f.runScript(Script_212421a51cfab87f:6)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.script.BaseScript.run(BaseScript.groovy:209)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
	at nextflow.cli.Launcher.run(Launcher.groovy:503)
	at nextflow.cli.Launcher.main(Launcher.groovy:657)

Environment

• Nextflow version: 24.04.3.5916
• Java version:

openjdk 17.0.10 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

• Operating system: docker of nextflow/nextflow:latest
• Bash version:

GNU bash, version 5.2.15(1)-release (x86_64-amazon-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[StefanBabukov]

Also experiencing the same issue