Erlang Error - Monotonic time stepped backwards - rabbit ouage

[AxoShachar]

Describe the bug

Running rabbit on Ubuntu using docker with rabbit version of 3.11
After several days of uptime, the rabbit container just collapsed and exited.
I can see the last logs contained a pretty clear Erlang error - Monotonic time stepped backwards!

could not locate the core dump, but the error is clear, searched for solution and saw something that might work.
+C no_time_warp | single_time_warp | multi_time_warp:

• https://www.erlang.org/doc/apps/erts/time_correction.html#Time_Correction
• https://www.erlang.org/doc/man/erl.html

any ideas on how I can debug the issue, or prevent it in the future, could not find anything in the documentation.
I can update Rabbit to the latest version 3.12- but not sure it will help

Reproduction steps

1. not sure how to reproduce - it happens. randomly

Expected behavior

Rabbit should not collapse/add an option with a flag to address this kind of issue.

Additional context

No response

[michaelklishin]

RabbitMQ does not adjust your operating system clock and cannot be assumed to be responsible for these very rare events. RabbitMQ 3.11 and 3.12 both require Erlang 25 and thus both use the same modern time API elements.

NTP events can moved OS time backwards a tiny bit. Maybe such correction are logged in various OS logs.

RabbitMQ allows you to specify any runtime flags you want to use, including those related to monotonic time and the time warp mode.

A quick search reveals relevant settings in various operating system kernels, time warp handling in other data services, and many discussions of how Os clocks are not perfect. It is a very broad area to try to narrow down.

RabbitMQ allows you to adopt any setting that Erlang provides.

[michaelklishin]

The enforcement of monotonic time (at least by default) first shipped in 25.0 and was introduced in erlang/otp@debc443.

I could not find a single issue or PR that would point to any incorrectness in the runtime.

You can try using

+C no_time_warp	

to get a leas strict behavior. Except for things like stream retention and TTL, RabbitMQ nodes are not as sensitive to time warp.

Raft keeps track of sequences using a bunch of counters. Heartbeat tracking would not be affected by a single ms stepping backwards since the recommended range is usually in the 15-60s range.

So both old and modern time warp defaults should work OK for RabbitMQ but one is obviously much stricter than the other, which can be an annoyance in some environments.

[Animalline]

I'm also having the same issue on my RabbitMQ cluster service. I have NTP services enabled on all of my servers, and I suspect this error was triggered while time syncing, but I didn't find the relevant logs of system time modifications, so I can't fully confirm that. In this case, should I disable the NTP service or configure no_time_warp mode, which really works?

[michaelklishin]

Do I recommend disabling NTP above or do I recommend something else? Also, how can we answer the "which one really works" question since you are guessing as to what has happened in your system?

[jason-shi]

+C no_time_warp how to set this?,
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
I use this script to install...

[michaelklishin]

We cannot possibly respond to a comment without Erlang version information and the runtime flags used.

@jason-shi this is not the first time when we ask for details and receive no relevant information back. I therefore don't have anything to add to #8940 (comment).