Super weird - first message (only) rejected

[AdamNormanJenkins]

Hi Everyone,

I have a super weird error. I have a rabbit server with some queues and a configured dlq and I'm using spring JMS/Rabbit. Everything is working great, messages get queued, the message are dequeued and processed and when I throw an AmqpRejectAndDontRequeueException it goes to the DLQ just fine for analysis. With one very unusual exception - when I startup my app, everything startup fine (and RabbitMQ has been running for days, so only my app is restarted) and the application starts handling requests but the very first, and ONLY the first, message sent to rabbit goes directly to the DLQ and doesn't trigger the listener at all (I have debug breakpoints)! The second, third, fourth etc etc all work fine, just the first message. Nothing is showing up in either the app logs or the rabbit logs that would suggest what's happening and the dlq message gives the following reason:

x-first-death-exchange: my exchange
x-first-death-queue: connect
x-first-death-reason: rejected

Any thoughts? I'm really stumped here.

rabbitTemplate.convertAndSend(connectRoutingKey, message);

messages get dequeued with

@RabbitListener(queues = "${app.mq.queue.connect}", concurrency = "${app.mq.concurrency.connect}", containerFactory = "prefetchNoneRabbitListenerContainerFactory") public void recievedMessage(ProcessConnectionRequest request) throws ConnectorException { try { connectorService.registerAsInProgress(request.getConnectorConfigUuid()); connectorService.connect(request.getConnectorConfigUuid()); }catch(Throwable t){ log.error("!!!!ERROR PROCESSING CONNECTOR!!! [" + request.getConnectorConfigUuid() + "]", t); throw new AmqpRejectAndDontRequeueException("!!!!ERROR PROCESSING CONNECTOR!!! [" + request.getConnectorConfigUuid() + "]", t); } }

The contains factory is configured thus:

@Bean public SimpleRabbitListenerContainerFactory prefetchNoneRabbitListenerContainerFactory(ConnectionFactory rabbitConnectionFactory) { SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory(); factory.setPrefetchCount(0); factory.setMessageConverter(jsonMessageConverter()); factory.setConnectionFactory(rabbitConnectionFactory); return factory; }

and my yaml is configured thus:

rabbitmq: host: <host> port: 5671 username: <username> password: <password> virtual-host: / listener: direct: prefetch: 0 simple: prefetch: 0 ssl: enabled: true cache: connection: mode: CONNECTION

[lukebakken]

Hello, thanks for using RabbitMQ.

It would be great to know the version of all the software you're using:

• RabbitMQ and Erlang
• Your JMS client
• Java

but the very first, and ONLY the first, message sent to rabbit goes directly to the DLQ

Is this message unrouteable or different from other messages in any way?

If you can reproduce this issue every time, the fastest way to help us help you is provide a complete project that we can clone, compile, and run to see the issue.

[AdamNormanJenkins]

ok, I've absolutely narrowed it down to how I created my dead letter queue so I thought I'd keep the post going in case anyone else hits this problem. It seems I'm doing something bonehead with my DLQ configuration because I unbound the dlq routing key and deleted the main message queue and recreated it with no DLQ support (so just a normal simple queue) and it all started working again. So, here is my DLQ configuration (and remember, I'm not throwing any exceptions or anything in my producer or consumer, just publishing to the message queue and it's being marked as rejected and routed directly to the DLQ on the first message only and if I shovel it back to the main queue, or my app publishes again, it all works fine. So my DLQ config is thus:

Exchange: "isaidusaid"
type: x-delayed-message
Routing: routingKey="analysis" -> queue: "analysis
Queue: "analysis"
type: classic
x-dead-letter-exchange: dlq
x-dead-letter-routing-key: dlq.analysis
durable: true

Exchange: "dlq"
type: direct
Routing: routingKey="dlq.analysis" -> queue: "dlq.analysis
Queue: "qlq.analysis"
type: classic

[AdamNormanJenkins]

ok....I just deleted and recreated the "analysis" queue, and everything is now working fine (for now, I'll keep an eye on it and post back if it stops working). I do remember reading a post somewhere over the last week of investigating that said that the dlq and routing needs to be created and bound BEFORE creating the message queue but I can't remember where I saw it and I can't speak to it's accuracy or whether this fix will sustain itself long term but for now it seems that recreating the queue solved the problem

[AdamNormanJenkins]

Edit: It's randomly started failing again - the trigger was I clicked "purge messages" on my DLQ and the issue came back immediately, the only other thing I did was restart my app so current theory is that something is happening in the 'purge messages' part of the DLQ that's messaging things up for the next publish or something is happening on shutdown of my app that's messaging things up for the next publish.

Edit: The issue below was my fault - in deleting and recreating all the queues I accidentally deleted the exchange routing dlq.connect which is why it wasn't going to DLQ, after adding it back it all seems to be working correctly (I'll keep an eye on it and let you know).

For those playing at home the fix seems to be 'recreate your queues' - I guess that's the JMS version of 'turn it off and on again'. So we'll see if this is a stable fix and if the problem comes back or not - if you don't see anymore posts from me then recreating the queues fixed the issue :)

Cheers of the help, Adam

ugh - I recreated ALL my queues, and the issue went away and the messages were automatically delivered, but then when I added a test exception in the spring message listener (which is supposed to, and previously did, route to dlq) the message just evaporated - no routing anywhere (let alone dlq).

@RabbitListener(queues = "${app.mq.queue.connect}",
concurrency = "${app.mq.concurrency.connect}",
containerFactory = "prefetchNoneRabbitListenerContainerFactory")
public void recievedMessage(ProcessConnectionRequest request) throws ConnectorException {
throw new AmqpRejectAndDontRequeueException("test rejection");
}

[AdamNormanJenkins]

restart does not fix the problem
can confirm that deleting and recreating the message queue (without modifying the dlq at all) fixes the problem
Attempting to get steps to reproduce now

[AdamNormanJenkins]

can confirm that it's NOT caused by the purge message button
can confirm that recreating the message queue fixes the problem
the only thing I can think is that in dev mode I'm starting and stopping the consumer a lot so maybe the queue is getting corrupted somehow?

[lukebakken]

@AdamNormanJenkins - after reading your last few messages, it's clear there are quite a few moving parts in your environment. Since this issue can be reproduced every time, you should be able to provide what we need to also reproduce it:

• An export of the necessary RabbitMQ entities (via definition export)
• A JMS program that publishes messages the same way as you do

You may have found a bug, but more than likely there is something in your code or setup that needs to be fixed. Neither I nor anyone else on the RabbitMQ team can spend time speculating about what the issue is.