'Timeout on blocking read' when sending messages

[cristibozga]

We migrate from old azure servicebus sdk to the new one and we have a couple of of problems after the first deployment. Here's one of them:
We get several errors with the message:
...
Caused by: java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:123)
at reactor.core.publisher.Mono.block(Mono.java:1766)
at com.azure.messaging.servicebus.ServiceBusSenderClient.sendMessage(ServiceBusSenderClient.java:266)
....

One timestamp: "@timestamp":"2025-01-06T15:01:40.831Z"
queue: tos_trace_queue
namespace: sbus-nonprod-sprint-main-pextnk

Sdk version: 7.17.7
Java version: 17

Here's the setup of the sender: 
public ServiceBusSenderClient prepareSender(String queueName) {
        return new ServiceBusClientBuilder()
                .connectionString(....)
                .retryOptions(RETRY_OPTIONS)
                .sender()
                .queueName(queueName)
                .buildClient();

    }

RETRY_OPTIONS are created using this code:

    Long TIMEOUT = 2000L;
    AmqpRetryOptions RETRY_OPTIONS=prepareRetryOptions();

    ServiceBusSenderClient prepareSender(String entityName);


    static AmqpRetryOptions prepareRetryOptions() {
        AmqpRetryOptions retryOptions = new AmqpRetryOptions();
        retryOptions.setMode(AmqpRetryMode.FIXED);
        retryOptions.setTryTimeout(Duration.ofMillis(TIMEOUT));
        return retryOptions;
    }

Where from the 10400000000 NANOSECONDS timeout comes and how can we avoid it?

Thanks,
Cristi

C:lone: Azure/azure-sdk-for-java#44688

[cristibozga]

New example of error we got today:
{"@timestamp":"2025-01-21T15:46:04.019Z","@Version":"1","message":"Exception processing message on queue WebhookNotificationBookingEventAcceptedQueue. Operation USER_CALLBACK","logger_name":"com.alpega.azure.servicebus.listener.handler.ServiceBusQueueErrorConsumer","thread_name":"boundedElastic-195","level":"ERROR","level_value":40000,"stack_trace":"com.azure.messaging.servicebus.ServiceBusException: An exception occurred while invoking the handler method; nested exception is org.springframework.messaging.MessageDeliveryException: Failed to send message to MyMessageChannel[MyMessageChannel@4ce6ec7]; nested exception is java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS, failedMessage=GenericMessage [payload={"payload":{"id":"GNFD00002142","reference":"Ref264128","type":"BOOKING","start":"2025-02-04T09:00:00Z","end":"2025-02-04T09:30:00Z","locationType":"PICKUP","groupName":"Group_0388140886197494","levelName":"Level_071907","externalApplication":"SLOT_V2","externalApplicationReference":null,"location":{"id":"A-b3juzu","name":"LisabonAddress03_2","code":null,"company":{"id":"PGMAI0OV","name":"Slot_SH1-ATSP048"},"address":null},"parties":[],"owners":[],"carriers":[{"id":null,"name":"OTHER"}]},"involvedCompanies":[{"businessId":null,"subscriptionType":"BOOKING_CARRIER","targetId":null},{"businessId":"PGMAI0OV","subscriptionType":"BOOKING_LOCATION_OWNER","targetId":"A-b3juzu"}],"eventType":"BOOKING_CANCEL","eventDate":"2025-01-21T15:45:53.404Z"}, headers={CorrelationId=00-5f433440e7ac69d433d6669f8352ae13-a855504e39a596ef-00, id=5c7eabf6-4cf6-c138-1d8c-6ae6cd5f0973, user=SPU-9zlhgp, contentType=application/json, timestamp=1737474353613}]\n\t....
Caused by: java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS\n\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:123)\n\tat reactor.core.publisher.Mono.block(Mono.java:1766)\n\tat com.azure.messaging.servicebus.ServiceBusSenderClient.sendMessage(ServiceBusSenderClient.java:266)\n\t...

Namespace: sbus-rnd-dev-main-qzlcup
Queue: SendWebhookNotificationQueue

Note: the problem appeared 2 times. First time at "2025-01-21T15:46:04.016Z

[cristibozga]

New test don on 10/03.
Got some errors in sbus-rnd-dev-main-qzlcup.servicebus.windows.net
Example:

{"@timestamp":"2025-03-10T21:45:58.526Z","@Version":"1","message":"Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.messaging.MessageDeliveryException: Failed to send message to MyMessageChannel[MyMessageChannel@1849ffe7]; nested exception is java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS, failedMessage=GenericMessage [payload={"type":"UPDATED","payload":{"locationBusinessId":"AT_De_LCTN_02cf044f-2d6a-4c4f-8c80-00c7b9989e2c"}}, headers={CorrelationId=00-7489f41177fd4babc1e7c9eea1b4ce2a-4994fa4eacdc2723-01, id=96deac66-3dc6-7913-3ee0-2324f244d236, user=AT_De_USR_c6e2f5ed-c5e4-4083-9b86-9fdb, contentType=application/json, timestamp=1741643148119}]] with root cause","logger_name":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]","thread_name":"http-nio-8080-exec-3","level":"ERROR","level_value":40000,"stack_trace":"java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS\n\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:123)\n\tat reactor.core.publisher.Mono.block(Mono.java:1766)\n\tat com.azure.messaging.servicebus.ServiceBusSenderClient.sendMessage(ServiceBusSenderClient.java:266)\n\tat com.alpega.azure.servicebus.sender.MyMessageChannel.doSend(MyMessageChannel.java:48)

Queue:UrgentInstructionsQueue

{"@timestamp":"2025-03-10T22:18:29.013Z","@Version":"1","message":"Unhandled exception from message handler method","logger_name":"com.alpega.azure.servicebus.listener.handler.ServiceBusQueueMessageHandler","thread_name":"receiverPump-8","level":"ERROR","level_value":40000,"stack_trace":"org.springframework.messaging.MessageDeliveryException: Failed to send message to MyMessageChannel[MyMessageChannel@3de6b4a6]; nested exception is java.lang.IllegalStateException: Timeout on blocking read for 10400000000 NANOSECONDS\n\tat org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:149)\n\tat org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:125)\n\tat org.springframework.messaging.core.GenericMessagingTemplate.doSend...
Queue: BookingInputQueueFIFO