MQTT bad topic name crash

[petervanalst]

Describe the bug

A device is able to send a message with unsupported characters in topic name that will crash RabbitMQ on handling the queue.
I cannot get rid of the message by trying to consume it, RabbitMQ crashes. Trying to get and/or remove it via management frontend will crash with 500 error.
Deleting the queue, and let it be created again looks the only way to resume operation.

Reproduction steps

Problematic device is one of about 2k devices that are connected. Others can have unparseable messages but will not crash RabbitMQ.

Expected behavior

RabbitMQ needs to reject messages with unsupported characters in topic name.

At least give a way to drop single message without affecting other messages in queue.

Additional context

Using rabbitmq:3.12.3-management with MQTT plugin, clustered using three nodes in separate docker environments.

Crash log

[mkuratczyk]

One clarification: these logs suggest a connection crash, not a RabbitMQ crash. I supposed RabbitMQ actually keeps handling other connections, correct?

Do you know what these characters are? I guess what we see in the logs is not what they really were.

[michaelklishin]

According to the log, it's an exception on a single connection, the RabbitMQ node should still be up.

You can purge the queue or set a TTL of 0 for messages on it using a policy to make RabbitMQ itself delete the message.

Without any details about what the unsupported value is, all I can think of is #9059, a fix for which has shipped in 3.12.4.
In fact, rabbit_mqtt_packet:serialise_utf in the stack trace suggests that it may indeed be a variation of #9059.

[michaelklishin]

OK, I now see the difference: #9059 dealt with LDAP queries specifically, whether in the context of MQTT or not. This is in an MQTT connection parser.

[michaelklishin]

@petervanalst can you have a guess as to what value and in what MQTT PUBLISH field was used to trigger this? We need to find a way to reproduce this behavior.

[petervanalst]

Yes the consumer connection fails. RabbitMQ keeps handling other connections.

But the consumer cannot resume operation with the specific queue, the connection will keep crashing until the bad message is gone.
What will not happen before i manually delete the queue and let it be remade by the consumer.

The device sent a corrupted version of a message with topic: "sva/86995103114911/alarm/strhgw", having a numeric value message of: 0, or: 1.
(The device ID is always in the topic, the log shows more corrupted device IDs at "atm_cache".)

The bad device is blocked and being replaced, i do not have a packet capture. Maybe i can find out more by looking at the good devices.

[michaelklishin]

@petervanalst sorry but nothing in "sva/86995103114911/alarm/strhgw" or 0 or 1 seems to be incorrect Unicode, in fact, these are all ASCII range characters.

Or your point was that the device somehow corrupted the data it self so it was not a correct sequence of bytes? Then the best RabbitMQ could do would be to close it with a more concise and specific error logged.

[ansd]

I suspect the same behaviour to happen in 3.11.x, but not in the latest 3.13 beta.
In a certain sense, it's great that the connection crashes, otherwise you would not have found out about the invalid topic name. Topic names in MQTT must be a UTF-8 encoded string.

We could remove that specific strict check (checking if a topic name is valid UTF-8 when sending a PUBLISH packet to the client) for 3.12 if we wanted.

The proper solution here would be to optionally introduce more checks when RabbitMQ receives MQTT packets: upon parsing packets, RabbitMQ could check that all data is valid according to the spec, and otherwise either terminate the connection or if this case would have happened in MQTT 5.0, RabbitMQ could reply with a PUBACK and reason code Topic Name invalid.

The AMQP 1.0 plugin for example defines a protocol_strict_mode configuration setting. We could add a similar setting that's disabled by default. If an operator enables such a strict mode, performance will decrease somewhat.