Internal error while connecting to Rabbit cluster from java application #8831
-
Describe the bugThis day we have found our test cluster fallen apart with network partition. All three nodes lost others. So it was a time to test upgrades and recovery. After restart nodes find others and cluster returned to operational state. On the server side logs are so on, gigabytes of. I was unable to understand a word except erlang files. It there something to extract useful knowledge and fix ? Reproduction stepsdon't know the roots of the problem, so don't know how to reproduce Expected behaviorsmooth and painless operation Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Answered by
michaelklishin
Jul 11, 2023
Replies: 2 comments 4 replies
-
|
The exception says that (probably all, or all on a single connection) channel received a message (an Erlang message) that had the structure it does not know how to handle. The function is In modern versions, these events are handled by queues, streams, connections, MQTT connections but My best hypothesis is that this was on an upgraded node and the messages originate from 3.11 nodes. I could not find any obviously related changes in the PRs for 3.12. Can you privately share |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
It is nothing special in definitions. |
Beta Was this translation helpful? Give feedback.
-
|
To summarize, we have tried two paths. 3.11.13-> 3.11.19->3.12.1 and 3.11.13->3.12.1 . Both were successful this time. There must be something special. |
Beta Was this translation helpful? Give feedback.
-
|
@KazzmanK wait, I know what this may be. Note that the 3.12.0 release notes explicitly say that you can perform rolling upgrades starting with |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Oh, as always, RTFM. Should we expect any consequences now? Or all troubles are gone after restarts ? |
Beta Was this translation helpful? Give feedback.
-
|
Once all nodes run the same 3.12 patch version, this specific problem (and well, this class of problems) should no longer be an issue. It's an inter-node communication format change. Except for scenarios where we know older versions won't be compatible (those are hidden behind feature flags, and every so often batches of feature flags graduate to required), not every environment will |
Beta Was this translation helpful? Give feedback.
Answer selected by
michaelklishin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@KazzmanK wait, I know what this may be. Note that the 3.12.0 release notes explicitly say that you can perform rolling upgrades starting with
3.11.17and you were attempting it from3.11.13. This is due to the internal Raft and QQ change that was backported for safety, but is not present in3.11.13.