S3AsyncClient#selectObjectContent documentation incorrect

[martijnhoekstra]

Describe the issue

The documentation indicates

Returns:
A Java Future containing the result of the SelectObjectContent operation returned by the service.

but the implementation returns CompletableFuture<Void>. In addition, the second parameter is undocumented. That makes it unclear how the API should be used, or indeed how to get the result of the operation.

Links

https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3AsyncClient.html#selectObjectContent(software.amazon.awssdk.services.s3.model.SelectObjectContentRequest,software.amazon.awssdk.services.s3.model.SelectObjectContentResponseHandler)

[martijnhoekstra]

note: perhaps it's actually/also the implementation that's wrong, and it should return a Future<Result> rather than a Future<Void>? It for sure it's not easy to get the results from the ResponseHandler

[debora-ito]

CompletableFuture<Void> is correct, it's not a bug.

SelectObjectContent is an event stream operation, so you need to use the SelectObjectContentResponseHandler to handle the response stream.

I see we don't have a S3 SelectObjectContent example in our Developer Guide, I'll work with our Docs Team and add one. In the meantime, take a look at the code in our Integration Tests and see if it's helpful -
https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/it/java/software/amazon/awssdk/services/SelectObjectContentIntegrationTest.java#L51

[martijnhoekstra]

I see, I would have expected the returned value to hold some sort of API for streaming the response -- sort of like the V1 sync client does.

The examples do help, though it's a bit dishearting to see that the exposed API is so low level: Records message: Can contain a single record, partial records, or multiple records. Depending on the size of the result, a response can contain one or more of these messages. getting a stream of records out of the stream of messages is something I would have expected the SDK to do, not something to be done on the application level.

Documentation with an example that shows getting a stream of messages in CSV and JSON output would be very handy, since that's the intended use.

[namankr]

Thats true @martijnhoekstra !

The problem is, that you may get multiple RecordsEvent events, each containing an InputStream with a part of the data - so you have to iterate through events and join them:

here is how I am doing it in my application. Hope it will help you.

import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import software.amazon.awssdk.core.async.SdkPublisher;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.*;

import java.io.InputStream;
import java.io.SequenceInputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
@Service
@Slf4j
public class S3SelectExample {

   // private final AsyncS3 asyncS3; // Assume this is your asynchronous S3 client

    @Autowired
    private S3AsyncClient asyncS3;

    public S3SelectExample(S3AsyncClient asyncS3) {
        this.asyncS3 = asyncS3;
    }

    public InputStream selectS3ObjectContent(SelectObjectContentRequest socRequest) {
        S3SelectObjectContentHandler handler = new S3SelectObjectContentHandler();
        CompletableFuture<Void> future = asyncS3.selectObjectContent(socRequest, handler);
        future.join();

        List<SelectObjectContentEventStream> receivedEvents = handler.getReceivedEvents();
        if (receivedEvents.stream().noneMatch(event -> event instanceof RecordsEvent)) {
            return InputStream.nullInputStream();
        } else {
            return receivedEvents.stream()
                    .filter(event -> event instanceof RecordsEvent)
                    .map(event -> ((RecordsEvent) event).payload().asInputStream())
                    .reduce(InputStream.nullInputStream(), SequenceInputStream::new);
        }
    }

    private static class S3SelectObjectContentHandler implements SelectObjectContentResponseHandler {
        private SelectObjectContentResponse response;
        @Getter
        private final List<SelectObjectContentEventStream> receivedEvents = new ArrayList<>();
        private Throwable exception;

        @Override
        public void responseReceived(SelectObjectContentResponse response) {
            this.response = response;
        }

        @Override
        public void onEventStream(SdkPublisher<SelectObjectContentEventStream> publisher) {
            publisher.subscribe(event -> receivedEvents.add(event));
        }

        @Override
        public void exceptionOccurred(Throwable throwable) {
            this.exception = throwable;
        }

        @Override
        public void complete() {
            // No operation
        }

    }
}

[debora-ito]

An example is now live in our Code Examples section - search for "SelectObjectContent".

Will go ahead and close this. Feel free to reach out if you have any other question.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.