NO_PROXY environment variable are not respected by S3CrtAsyncClient

[eugene-deng]

Describe the bug

S3CrtAsyncClient doesn't respect the values in the NO_PROXY environment variable

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The client should not establish connections to the host included in the NO_PROXY variable

Current Behavior

Once http_proxy and https_proxy are set, NO_PROXY env var or adding no proxy hosts to the S3CrtProxyConfiguration doesn't work and the client will try to establish a proxy-based connection.

Reproduction Steps

1. install localstack

2. copy the

//usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS software.amazon.awssdk:s3:2.31.28
//DEPS software.amazon.awssdk.crt:aws-crt:0.38.1
//DEPS ch.qos.logback:logback-classic:1.4.11 // For logging

package com.aws.clients;

import java.net.URI;
import java.util.concurrent.CompletableFuture;

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncClient;
import software.amazon.awssdk.services.s3.model.ListBucketsRequest;
import software.amazon.awssdk.services.s3.model.ListBucketsResponse;

public class S3CrtAsyncClientTest {

    public static void main(String[] args) {

        S3AsyncClient s3Client = S3CrtAsyncClient.builder()
                .credentialsProvider(DefaultCredentialsProvider.create())
                .endpointOverride(URI.create("http://localhost:4566"))
                .region(Region.US_EAST_1)
                .build();

        // Perform an S3 operation
        CompletableFuture<ListBucketsResponse> future = s3Client.listBuckets(ListBucketsRequest.builder().build());

        // Handle the response
        future.whenComplete((response, exception) -> {
            if (exception != null) {
                System.err.println("Failed to list buckets: " + exception.getMessage());
            } else {
                System.out.println("Buckets: " + response.buckets());
            }
        }).join();

        // Close the client
        s3Client.close();
    }
} 

3. running http_proxy="http://localhost:8888" https_proxy="http://localhost:8888" no_proxy="localhost" jbang src/main/java/com/aws/clients/S3CrtAsyncClientTest.java will throw error

http_proxy="http://localhost:8888" https_proxy="http://localhost:8888" no_proxy="localhost" jbang src/main/java/com/aws/clients/S3CrtAsyncClientTest.java
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Failed to list buckets: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: socket connection refused. (SDK Attempt Count: 1)
Exception in thread "main" java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: socket connection refused. (SDK Attempt Count: 1)
        at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:135)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:152)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:113)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: socket connection refused. (SDK Attempt Count: 1)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        ... 25 more

Extra - I also tested with S3Client

//usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS software.amazon.awssdk:s3:2.31.28
//DEPS software.amazon.awssdk.crt:aws-crt:0.38.1
//DEPS ch.qos.logback:logback-classic:1.4.11 // For logging

package com.aws.clients;

import java.net.URI;

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;

public class S3ClientTest {

    public static void main(String[] args) {
        S3Client s3Client = S3Client.builder()
                .endpointOverride(URI.create("http://localhost:4566"))
                .region(Region.US_EAST_1)
                .credentialsProvider(DefaultCredentialsProvider.create())
                .build();

        s3Client.listBuckets().buckets().forEach(bucket -> {
            System.out.println(bucket.name());
        });
    }
}

It works

http_proxy="http://localhost:8888" https_proxy="http://localhost:8888" no_proxy="localhost" jbang src/main/java/com/aws/clients/S3ClientTest.java
[jbang] Resolving dependencies...
[jbang]    software.amazon.awssdk:s3:2.31.28
[jbang]    software.amazon.awssdk.crt:aws-crt:0.38.1
[jbang]    ch.qos.logback:logback-classic:1.4.11
[jbang] Dependencies resolved
[jbang] Building jar for S3ClientTest.java...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
my-bucket

Possible Solution

I think the internal http client has some troubles parsing the env var as I see another issue it doesn't support wildcard
maybe the parsing has more issues

Additional Information/Context

No response

AWS Java SDK version used

2.31.28

JDK version used

21

Operating System and version

mOS Sequoia 15.4.1

[bhoradc]

Hello @eugene-deng,

Thank you for reporting the issue with the S3CrtAsyncClient and the NO_PROXY environment variable.

After analyzing the behavior, it seems that the AWS Common Runtime currently does not support the NO_PROXY environment variable. There is an open feature issue to track the implementation of this functionality - awslabs/aws-c-http#413.

I would encourage to upvote the mentioned issue to demonstrate your support for this feature. Upvoting the issue on GitHub may help to gauge community interest and prioritize the implementation of NO_PROXY support in the AWS Common Runtime.

Regards,
Chaitanya

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.