OOM WebClient Get Request with Body after couple hours

[LizzyMiao]

I have googled and tried with GPT, but no luck, so asking in this place, if not appropriate I will delete it.
I am using webClient in my springboot app like below code shows, after several hours later, there would be errors like "4194304 bytes of direct buffer memory", also pasted below...

The error all happens to one fixed service which usually returns around 20M-30M response, other get cmd works just fine...

I use Spring-webflux-5.3.31 + spring boot 2.17.18, openJdk 17.0.8_1

    private void initializeWebClient() {
        log.info("Initializing released webClient");
        ConnectionProvider connectionProvider = ConnectionProvider.builder("pool")
                .maxConnections(100)
                .pendingAcquireTimeout(Duration.ofSeconds(60))
                .build();

        ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
                .codecs(configuer -> configuer.defaultCodecs().maxInMemorySize(500 * 1024 * 1024))
                .build();

        HttpClient httpClient = HttpClient.create(connectionProvider)
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 100000)
                .doOnConnected(connection -> {
                    connection.addHandlerLast(new ReadTimeoutHandler(100000, TimeUnit.MILLISECONDS));
                    connection.addHandlerLast(new WriteTimeoutHandler(100000, TimeUnit.MILLISECONDS));
                });

        this.webClient = org.springframework.web.reactive.function.client.WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(httpClient))
                .exchangeStrategies(exchangeStrategies)
                .build();
    }

    public String get(String url, Map<String, String> headers) {
        String response = this.webClient.get()
                .uri(url)
                .headers(httpHeaders -> headers.forEach(httpHeaders::add))
                .retrieve()
                .onStatus(HttpStatus::isError,
                        clientResponse -> clientResponse.bodyToMono(String.class)
                                .flatMap(errorMessage -> Mono.error(new RuntimeException("Error: " + errorMessage))))
                .bodyToMono(String.class)
                .retryWhen(Retry.fixedDelay(MAX_RETRIES, RETRY_DELAY)
                        .doBeforeRetry(retrySignal -> log.info("Retrying... attempt: {}", retrySignal.totalRetries() + 1)))
                .doOnError((throwable) -> {
                    log.error("Error happened for get: {}, detail error: {}", url, throwable.getMessage());
                    throw new RuntimeException("Error happened for get " + url, throwable);
                })
                .block();

        return response;
    }

Caused by: java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 1070302774, limit: 1073741824)
at java.base/java.nio.Bits.reserveMemory(Unknown Source)
at java.base/java.nio.DirectByteBuffer.(Unknown Source)
at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:139)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)

[bclozel]

Thanks for getting in touch.

This might be a memory leak in your application, or in Spring itself. You can use the spring.netty.leak-detection=PARANOID property and run your application to find the source.
Unfortunately, we won't be able to assist you more since Spring Framework 5.3.x and Spring Boot 2.7.x are out of OSS support; so you'll need to upgrade to a supported version or consider commercial support. The version you are using is vulnerable to several important CVEs.

[LizzyMiao]

thank you!