Random PRECONDITION_FAILED error with timeout

[simakji]

Describe the bug

Hello everyone,

during our testing we have been getting a few times per hour these errors from RMQ with load of 5000 messages/min:

PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 15000 ms. This timeout value can be configured, see consumers doc guide to learn more

As you can see from the error message, we decreased timeout to 15s to see the error more often. We also got pcap by tcpdump, where we see that all consumed messages was successfully acknowledged by Basic.Ack, also there are no issue on TCP level. After we got the ERROR, TCP connection was closed in standard way.

Setup:
Client side:

• JVM based app
• used library: com.rabbitmq:amqp-client:5.18.0

Server side:

• 7 node cluster,
• RMQ 3.12.2,
• Erlang 25.1.2,
• standard non HA queues,
• 2 consumers,
• prefetch count 2,
• 15s consumer timeout.

Reproduction steps

1. Make the setup described above with load of 5000 messages/min.
2. Consume with manual acknowledgement.
3. Observe PRECONDITION_FAILED - timeout errors

Expected behavior

Do not receive timeout on already acknowledged messages.

Additional context

We would appreciate to fix this, since after the error we have to do a new connections which slows us down.

[michaelklishin]

You have not provided any evidence of a bug in RabbitMQ.

We cannot suggest anything without full logs. PRECONDITION_FAILED is used in a number of contexts, specifically in delivery acknowledgement timeouts which modern nodes enforce.

If your consumer does not acknowledge a message in the configured time period, this is how the server forcefully closes its channel. The docs explain many details but the most important thing to understand is that the timeout is a guard rail, not a way to implement timeouts for your applications.

[michaelklishin]

Taking a traffic capture is by far the best way to see what your consumers really do, in particular, when and how they acknowledge deliveries. The problem is on your consumers' side.

[simakji]

Here is the pcap.
rmq-02.pcap.gz

[lukebakken]

How did you take this traffic capture? On what machine, and what command was run, exactly?

[simakji]

The pcap was captured by
tcpdump -i any port <port> -w filename.pcap -v
on client side machine. Then stripped by Wireshark.

[lukebakken]

@simakji - as @michaelklishin said, your consumers are not acknowledging messages correctly.

The simplest way for us to help you is to provide code that reliably reproduces what you observe. I should be able to clone a git repository, compile it, and run it to see the behavior.

[simakji]

I will try to do it. We are getting that error on our testing environment (VM based), so I am not sure if it will be reproducible on localhost.