Correlation between src-prefetch-count and src-delete-after in dynamic shovel?

[rorytorneymf]

Hello,

I am using RabbitMQ 3.11.11 / Erlang 25.3

I am creating 1 source queue (my-src-queue) and 1 destination queue (my-dst-queue), and populating my-src-queue with 100000 messages, like this:

try(final Connection connection = connectionFactory.newConnection()) {
    final Channel channel = connection.createChannel();

    channel.queueDeclare("my-src-queue", true, false, false, Collections.emptyMap());
    channel.queueDeclare("my-dst-queue", true, false, false, Collections.emptyMap());

    Consumer consumer = new DefaultConsumer(channel) {
        @Override
        public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
            String message = new String(body, "UTF-8");
            System.out.println("Received message: " + message);
        }
    };

    channel.basicConsume("my-dst-queue", true, consumer);

    final AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
            .contentType("application/json")
            .deliveryMode(2)
            .priority(1)
            .build();
    for (int i = 0; i < 100000; i++) {
        channel.basicPublish("", "my-src-queue", properties, gson.toJson(new Object()).getBytes(StandardCharsets.UTF_8));
    }

    Thread.sleep(6000000);
}

I am then creating a dynamic shovel to move 1000 messages from my-src-queue to my-dst-queue:

curl --location --request PUT 'http://localhost:49231/api/parameters/shovel/%2f/testshovel' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Z3Vlc3Q6Z3Vlc3Q=' \
--data '{
  "component": "shovel",
  "name": "testshovel",
  "value": {
    "src-uri":  "amqp://",
    "src-queue":  "my-src-queue",
    "dest-uri": "amqp://",
    "ack-mode": "on-confirm",
    "dest-queue": "my-dst-queue",
    "src-delete-after": 1000
  }
}'

When 1000 messages have been moved and the shovel has been automatically deleted (because of the src-delete-after property), I then create the above shovel again to move another 1000 messages and repeat this about 10 times.

The problem is I am seeing quite a lot of 'Redelivered' spikes on the charts for my-src-queue (around 200 /s), and I don't know why:

I repeated the test but set src-prefetch-count to 10 this time:

curl --location --request PUT 'http://localhost:49231/api/parameters/shovel/%2f/testshovel' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Z3Vlc3Q6Z3Vlc3Q=' \
--data '{
  "component": "shovel",
  "name": "testshovel",
  "value": {
    "src-uri":  "amqp://",
    "src-queue":  "my-src-queue",
    "dest-uri": "amqp://",
    "ack-mode": "on-confirm",
    "dest-queue": "my-dst-queue",
    "src-delete-after": 1000,
    "src-prefetch-count": 10
  }
}'

and I see the 'Redelivered' rate drop to about 2 /s, which is better:

I am wondering if this is the right thing to do, or if there was any advice about what the src-prefetch-count should be set to in order to reduce these redeliveries (my shovels are created dynamically, so won't always be shoveling 1000 messages).

Many thanks!

[michaelklishin]

In order to reduce the redeliveries you should not be using Shovels that self-delete: they will not pay attention to any unacknowledged messages. You can delete Shovels programmatically.

You already use "on-confirm" for acknowledgement mode, so it must be the current limitations of the self-deletion condition.

[rorytorneymf]

Thanks @michaelklishin. Assuming I continue using shovels that self-delete, setting src-prefetch-count=10 on an self-delete shovel (src-delete-after=1000) seemed to reduce the redeliveries substantially, would you not recommend that approach? Is there any downside to setting src-prefetch-count to a low value?

[michaelklishin]

Higher prefetch means higher throughput up to a point. If 10 works for your needs, go for it.

Perhaps a pause setting before auto-deletion would help the last "batch" of messages to get through,
I'd need to take a look at the code and try to reproduce.