MQTT Connection not stable after Upgrading to 3.12-rc2

[JanEggers]

Describe the bug

We use MQTT Explorer to monitor the events that are processed by the broker. with version 3.11.* it is running fine.

After updating to 3.12-rc2 we get client disconnected errors.

Reproduction steps

publish events

Expected behavior

stable MQTT Connection

Additional context

No response

[JanEggers]

rabbit logs.txt

[michaelklishin]

I'm afraid "publish events" is not a set of steps to reproduce. Our tests and benchmark do publish messages.

Please provide a traffic capture or a way to reproduce with mosquitto_pub or a similar tool where a set of publisher properties can be easily shared.

[kjnilsson]

2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>        [{rabbit_mqtt_processor,deliver_one_to_client,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>             [{{resource,<<"/">>,queue,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>               <0.4546.0>,undefined,false,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>               {basic_message,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   {resource,<<"/">>,exchange,<<"amq.topic">>},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   [<<"plant.v1.M3.BCD423.rev.fillStateChangedEvent">>],
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   {content,60,none,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                       <<80,64,16,97,112,112,108,105,99,97,116,105,111,110,47,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                         106,115,111,110,2,0,0,0,0,100,103,141,186>>,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                       rabbit_framing_amqp_0_9_1,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                       [<<"{\"plantIdentificationCode\":\"M3/BCD423/rev\",\"isFullSensorTriggered\":false,\"numberOfCarriers\":20,\"maxNumberOfCarriers\":1174,\"numberOfEmpties\":0,\"numberOfCarriersWithPayload\":20,\"numberOfCarriersWithOrder\":0,\"trackLength\":30000,\"trackLengthOccupied\":1130}">>]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   <<31,230,178,158,209,240,53,221,100,60,64,5,227,237,58,21>>,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                   true}},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>              false,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>              {state,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  {cfg,#Port<0.282>,mqtt311,true,undefined,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      {resource,<<"/">>,exchange,<<"amq.topic">>},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      undefined,false,none,<0.680.0>,flow,none,10,<<"/">>,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      <<"mqtt-explorer-c5351d21">>,undefined,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      {192,168,10,131},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      1883,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      {192,168,10,130},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      53244,1684508087392,#Fun<rabbit_mqtt_reader.0.106886>},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  {rabbit_queue_type,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      #{{resource,<<"/">>,queue,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                            <<"mqtt-subscription-mqtt-explorer-c5351d21qos0">>} =>
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                            {ctx,rabbit_classic_queue,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                                {rabbit_classic_queue,<0.4546.0>,#{},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                                    #{<0.4546.0> => ok}}}}},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  #{},#{},1,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  #{<<"#">> => 0,<<"$SYS/#">> => 0},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  {auth_state,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      {user,<<"rabbit">>,[],
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                          [{rabbit_auth_backend_internal,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                               #Fun<rabbit_auth_backend_internal.3.114557357>}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                      #{<<"client_id">> => <<"mqtt-explorer-c5351d21">>}},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>                  registered,#{},0}],
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>             [{file,"rabbit_mqtt_processor.erl"},{line,1414}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {lists,foldl,3,[{file,"lists.erl"},{line,1350}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {rabbit_mqtt_processor,handle_queue_event,2,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>             [{file,"rabbit_mqtt_processor.erl"},{line,1345}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {rabbit_mqtt_reader,handle_cast,2,
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>             [{file,"rabbit_mqtt_reader.erl"},{line,134}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1123}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1200}]},
2023-05-19 16:54:51.167000+02:00 [error] <0.4541.0>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

[michaelklishin]

MQTT Explorer seems to be a graphical tool. @JanEggers how should I use it to reproduce?

[michaelklishin]

I tried a number of tools (Paho Python, Mosquitto) but could not reproduce.

However, thanks to @kjnilsson @ansd we understand what the root cause is, just not what kind of integration test would help.

[michaelklishin]

3.12.0-rc.3 includes #8254.

[JanEggers]

Thx will test again