messages remain unacknowledged

[FlaviuRadulescu]

Hi,

Would it be possible to give me some recommendations for the following problem?

I have some consumers, and each consumer needs to have in RAM a large file, as these large files take time to load in memory (like 10-15 seconds) I would prefer to deliver messages just to consumers that have the file in memory, as these files could not fit in the memory of all the consumer, I am having some consumers having a file, and other some other files, in different moments some of the files are replaced in memory so the consumer configuration is dynamic.

the way I have implemented this is that each consumer has a priority, the consumer with the file in RAM has an x-priority higher than the ones that don't have the file in RAM. All the consumers are connected to multiple queues, each queue being the file name.

What is happening is that all the consumers are connected to all the queues (direct connection without exchange), but some consumers from a queue have higher priority than others. So, in this way, the consumer with the file in memory gets the message so that the consumer without the file in memory doesn't receive it because these messages were already processed by other consumers.

The problem I have is that I am not sure the above is the right implementation, and the biggest problem is that for an unknown reason, some of the messages (a small percentage) remained unknowledge. because of this, some messages are lost because are not processed in a certain period (being stuck in the queue).

the implementation I have is something like this
$rabbitMqQueueConnection = rabbitmq connection

$rabbitMqQueueConnection->setQos();
while ($rabbitMqQueueConnection->isConsuming()) {

-> Check if we need to add a file in memory.
-> If the file needs to be added to memory we add it
-> If the file was added to memory, we disconnect from the queue and reconnect to the queue with a higher priority

$rabbitMqQueueConnection->wait(null, true); //non blocking
}

I am almost 100% sure (checked the code hundreds of times) that we are acknowledging the message in the callback, but I think the disconnection and reconnection from the while loop before the wait may somehow trigger the message to not be acknowledged to the backend.

So, I have a few questions:

1. is there a better implementation that I could do instead of connecting and disconnecting from the queues depending on what files are in memory?
2. is there a way to figure out at what moment or what is triggering the messages to not be acknowledged? (one thing to mention here, is that the code continues to work receiving messages from other queues. Also, I saw that when this happens the unknowledge messages are all on the same queue from the same server/connection
3. is there a way to check if the message is not acknowledged yet and do this before disconnecting from the queue to reconnect with a different priority?

Thank you so much, I will be very grateful if you can help me find and fix this issue

[michaelklishin]

Consumers that are not ready to handle deliveries (by this definition, do not yet/currently have the file loaded into RAM) arguably should not be subscribed. But this really depends on how often they may go into this state or out of it. If it is something that happens once a minute, or only on application boot, I'd suggest dynamically adding/removing bindings and registering/canceling the consumer, that would not really create a binding or consumer churn scenario. You don't have to close their channels or connection.

We cannot help you troubleshoot your consumers, neither clients libraries nor RabbitMQ itself acknowledge messages when manual acknowledgement mode is used, your applications do.

A delivery is only delivered to one consumer at a time, so as long as your consumer has the delivery, it cannot be delivered to (and acknowledged by) another consumer.