File descriptor use monitoring

[jtpsh]

Describe the bug

The file descriptor metric reported by rabbitmq_management and rabbitmq_prometheus is much lower than the reported count from rabbitmqctl status.

From our investigation (cmiiw), it looks like both management and prometheus plugins might be getting their FD count effectively like ls /proc/PID/fd | wc -l. On the other hand the output of rabbitmqctl status is getting FD count using file_handle_cache:info(). The output of the two are vastly different and the file_handle_cache:info() seems to be the more accurate one. This is a problem since if we rely on FD alert from prometheus, we won't be alerted when the FD limit is reached (which happened to our case). There were also no log entries indicating that the FD limit has been reached.

Here's the snap from our simple test earlier (see reproduction steps below):

# (first column is data source, second column is FD count)

Mon Jul 22 02:29:39 PM UTC 2024
file_handle_cache 0
rabbitmqctl status 0
/proc/1269599/fd 46
prometheus 45


Mon Jul 22 02:29:43 PM UTC 2024
file_handle_cache 0
rabbitmqctl status 0
/proc/1269599/fd 46
prometheus 45

-- snipped --
-- After creating enough queuees --

Mon Jul 22 02:37:13 PM UTC 2024
file_handle_cache 546
rabbitmqctl status 546
/proc/1269599/fd 52
prometheus 52


Mon Jul 22 02:37:17 PM UTC 2024
file_handle_cache 546
rabbitmqctl status 546
/proc/1269599/fd 53
prometheus 52

Reproduction steps

1. Configure a lower FD limit on the system, (we use 500 so we can easily reach the limit)

   # /etc/systemd/system/rabbitmq-server.service.d/limits.conf
   [Service]
   LimitNOFILE=500
   

2. Login to rabbitmq server and check the FD count from various sources

   RMQ_PID=123456
   while true; do 
     sleep 1;
     date;
     echo file_handle_cache $(/usr/sbin/rabbitmqctl --timeout 10 eval 'file_handle_cache:info().' --formatter json | jq -r '.value.total_used'); 
     echo rabbitmqctl status $(/usr/sbin/rabbitmqctl --timeout 10 status --formatter json | jq '.file_descriptors.total_used'); 
     echo /proc/$RMQ_PID/fd $(ls /proc/$RMQ_PID/fd | wc -l);
     echo prometheus $(curl -s localhost:15692/metrics | grep '^rabbitmq_process_open_fds' | awk '{print $2}'); 
     echo; echo;
   done
   

3. Run a consumer/producer continuously in the background to simulate connections. We use perftest for that

   java -jar perf-test-2.20.0.jar --uris amqp://admin:[email protected]:5672/myvhost -r 10 -q 1
   

4. Create enough queues to increase FD count

   echo qq-{001..500} | tr ' ' '\n' | awk '{ system("sleep 1"); print $1 }' | xargs  -I {} ./rabbitmqadmin -u admin -p 1123581321a --vhost myvhost declare queue name={}
   

5. Observe the output in Step 2. You will see that the FD count from file_handle_cache (and rabbitmqctl status) is increasing relative to the number of queues created.
   The output from ls /proc/PID/fd and prometheus is not increasing and almost stays the same.

   At some point, the count from file_handle_cache will reach the limit and rabbitmqctl status will start timing out.
   This is a good indication that the FD limit has been reached. If you open another connection using perftest, you will also get an error. The perftest in Step 3 should still be running at this point without any issues. Only new connections are affected.

6. Manually increase the limit using /usr/sbin/rabbitmqctl --timeout 10 eval 'file_handle_cache:set_limit(1024).'

7. New connections using perftest are now working again and so is rabbitmqctl status command.

Expected behavior

1. idk which is the correct one but from our tests, the FD count from file_handle_cache seems to be the more reliable one and should be reported instead by rabbitmq_management and rabbitmq_prometheus.
2. There should be a log entry when the FD limit is reached

Additional context

OS: Oracle Linux 9.3 using Redhat compatible kernel

uname -a
Linux <hostname> 5.14.0-362.24.1.0.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 2 08:32:36 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Tested RabbitMQ versions:

• rabbitmq-server-3.13.1
• rabbitmq-server-3.13.5

[michaelklishin]

file_handle_cache was removed from RabbitMQ for 4.0. We don't want every single component in RabbitMQ to depend on FHC, and FHC has its own issues that you are likely unaware of.

It is an OS-level metric and should be monitored outside of RabbitMQ.

[jtpsh]

Thank for the info, appreciate it!

[jtpsh]

@michaelklishin Thank you again for the insight. Just following up here so we can close this on our end.

1. How accurate is file_handle_cache and is it possible for it go "out of date"? For example, counting FDs in my dev server via lsof or traversing /proc/ would give a much closer value to file_handle_cache. However, in prod cluster where there is much activity the counts varies by a wide margin.
2. If 1 is correct, does that mean we could probably just set our FD limit so high it may be impossible to reach?

Thanks

[mkuratczyk]

If I'm not mistaken, the metric is accurate for what it does - counting file descriptors used by file_handle_cache, but while in the past, virtually all RabbitMQ components used the FHC, in the latest versions very few still do. For example quorum queues don't use FHC at all (they never have used it I believe), so files used by quorum queues are not accounted for in this metric at all.

Yes, these days you should just set a high limit and you're very unlikely to have to think about FDs ever again. You can monitor FD usage on the operating system level if you want, just to be sure. The FHC is going away - it was useful a long time ago, but things change and right now it's just a bottleneck.

[michaelklishin]

@jtpsh how can FHC possibly be more accurate than the kernel? Why do you assume that a newly declared queue will use a file handle for the entire duration of its lifetime, and why should that be equally true for all queue types?

You need to revisit your fundamental assumptions.

[michaelklishin]

There should be a log entry when the FD limit is reached

There already are, because in 3.13 surpassing the limit triggers a resource alarm. However, a log message won't be enough. What you really need is OS-level monitoring of file handles with alerts.

RabbitMQ is not particularly special when it comes to FD and socket file handle use, so any external monitoring tool can monitor those metrics just like it could monitor any other process.

[michaelklishin]

Finally, there are per-virtual host guardrails for environments where applications are known (or can be expected to) leak connections or queues.