How to disable max-content-length check for http client for particular API call ?

[vasanthegde]

When I am making HEAD call to an api, I am seeing io.micronaut.http.client.exceptions.ContentLengthExceededException
by deafult micronaut.http.client.max-content-length = 10MB
since this is int type and max value we can set to it is 2147483647 ( ie 2GB)

the API I am calling is for streaming file storage and these file size can be bigger than 2GB, in that how do I overcome this exception ?
is there anyway to increase it to beyond 2GB or disable it altogether ? any help would be appreciated.

client version : micronaut-http-client-core:3.4.4

[yawkat]

i don't think there's a way to increase the limit, but you can use a streaming response (eg dataStream) to disable it

[yawkat]

Flux<HttpResponse<String>> won't work, but Flux<String> should.

[vasanthegde]

thanks, I will try this out.

[m4nu56]

Hey, I'm curious did you manage to download files bigger than 2GB in the end ?
Could you share how you manage to do it ?

On my side I'm using the micronaut http client to do a very basic http download like:

HttpResponse<byte[]> result = httpClient.toBlocking().exchange(
                    HttpRequest.GET(url)
                            .bearerAuth(token)
                            .header(HttpHeaders.ACCEPT, MediaType.APPLICATION_OCTET_STREAM), byte[].class);

I could transform that to be reactive instead and that would get around the maxContentLength of 2GB issue with the micronaut netty http client ?

Thank's

[yawkat]

Correct

[m4nu56]

Ok thank's, I've replaced my code with the following and it seems to work well. Tests are green anyway :)

final byte[] body = httpClient.exchangeStream(
                            HttpRequest.GET(url)
                                    .bearerAuth(token)
                                    .header(HttpHeaders.ACCEPT, MediaType.APPLICATION_OCTET_STREAM))
                    .doOnNext(httpResponse -> {
                        LOGGER.debug("Received a new buffer of the file with HTTP Status: {} with length: {}", httpResponse.getStatus(), httpResponse.body().toByteArray().length);
                        if (httpResponse.getStatus() != HttpStatus.OK) {
                            throw new FileServiceException(String.format("Error downloading file \"%s\". Invalid response code [Code: %s, expected: 200]. Result: %s", url, httpResponse.getStatus().getCode(), httpResponse));
                        }
                    })
                    .doOnComplete(() -> LOGGER.info("Download file {} complete", url))
                    .doOnError(throwable -> LOGGER.error("Error downloading file {}: {}", url, throwable.getMessage()))
                    .collectInto(new ByteArrayOutputStream(), (bytes, byteBufferHttpResponse) -> bytes.write(byteBufferHttpResponse.body().toByteArray()))
                    .map(ByteArrayOutputStream::toByteArray)
                    .blockingGet();