rabbitmq_io_write_ops_total and rabbitmq_io_read_ops_total in environments that do not use classic queues

[lanzhiwang]

Describe the bug

In the rabbitmq 3.12.4 version, the quorum queues is used to produce and consume messages, and the plug-in is used to output monitoring indicators to Prometheus.

Among the monitoring indicators, rabbitmq_io_write_ops_total and rabbitmq_io_read_ops_total are always 0.
It looks like there are no disk IO operations.

In fact, data is written to the disk, and the size of the wal file is constantly growing. This can mean that the data is written to the disk.

$ ls -al ./
total 95236
drwxr-xr-x 3 polkitd input 107 Mar 28 17:41 .
drwxr-xr-x 3 polkitd input 71 Mar 28 17:11 ..
-rw-r--r-- 1 polkitd input 64000194 Mar 28 17:41 00000043.wal
-rw-r--r-- 1 polkitd input 23984472 Mar 28 17:41 00000044.wal
drwxr-xr-x 3 polkitd input 8192 Mar 28 17:41 2F_THRUFCAY7FKEVRD
-rw-r--r-- 1 polkitd input 5570 Mar 28 17:41 meta.dets
-rw-r--r-- 1 polkitd input 5910 Mar 28 17:18 names.dets

$ ls -al ./
total 95236
drwxr-xr-x 3 polkitd input 107 Mar 28 17:41 .
drwxr-xr-x 3 polkitd input 71 Mar 28 17:11 ..
-rw-r--r-- 1 polkitd input 64000194 Mar 28 17:41 00000043.wal
-rw-r--r-- 1 polkitd input 29078061 Mar 28 17:41 00000044.wal
drwxr-xr-x 3 polkitd input 8192 Mar 28 17:41 2F_THRUFCAY7FKEVRD
-rw-r--r-- 1 polkitd input 5570 Mar 28 17:41 meta.dets
-rw-r--r-- 1 polkitd input 5910 Mar 28 17:18 names.dets

I would like to ask why the above two monitoring indicators are always 0. Is it a problem with the monitoring plug-in itself or the mechanism of the quorum queues itself?

Reproduction steps

1. rabbitmq 3.12.4

2. rabbitmq config

luster_formation.peer_discovery_backend=rabbit_peer_discovery_k8s
cluster_formation.k8s.host=kubernetes.default
cluster_formation.k8s.address_type=hostname
cluster_partition_handling=pause_minority
queue_master_locator=min-masters
disk_free_limit.absolute=2GB
cluster_formation.randomized_startup_delay_range.min=5
cluster_formation.randomized_startup_delay_range.max=10
mnesia_table_loading_retry_timeout=30000
mnesia_table_loading_retry_limit=3
log.console=true
log.console.level=debug
log.connection.level=debug
log.channel.level=debug
log.queue.level=debug
log.mirroring.level=debug
log.federation.level=debug
log.upgrade.level=debug
log.default.level=debug
cluster_name=my-rabbitmq
total_memory_available_override_value=1717986919
channel_max=1000
cluster_keepalive_interval=10000
collect_statistics=none
collect_statistics_interval=5000
default_vhost=/
delegate_count=16
disk_free_limit.absolute=50MB
handshake_timeout=10000
heartbeat=60
max_message_size=134217728
mirroring_sync_batch_size=4096
mnesia_table_loading_retry_limit=10
mnesia_table_loading_retry_timeout=30000
num_acceptors.tcp=10
queue_index_embed_msgs_below=4096
raft.wal_max_size_bytes=64000000
vm_memory_calculation_strategy=allocated
vm_memory_high_watermark.relative=0.4
vm_memory_high_watermark_paging_ratio=0.5

Expected behavior

I would like to ask why the above two monitoring indicators are always 0. Is it a problem with the monitoring plug-in itself or the mechanism of the quorum queues itself?

Additional context

No response

[michaelklishin]

Quorum queues use a completely different storage "subsystem": the on in our Raft implementation. This also affects streams and Khepri.

They are available as metrics in a separate Grafana dashboard.

I'd not rely on RabbitMQ HTTP API metrics for something like disk I/O activity. Using infrastructure metrics is going to give you a more complete picture.

[lanzhiwang]

There are a few questions that need to be confirmed:

1. Is the problem described above a bug? If not, you can remove the bug label

2. Can all the available as metrics be obtained from the rabbitmq_prometheus plug-in, or are these metrics from other sources?

3. How to obtain the metrics under group node_persister_metrics, such as rabbitmq_detailed_io_read_ops_total and rabbitmq_detailed_io_write_ops_total, etc., I see that these metrics are only defined in the source code, but I have not found where they are obtained, for example, from /sys /proc of the os system. Obtained it?

[michaelklishin]

It will be considered a bug by some and just a stale metric by others. As a lot of old subsystems in RabbitMQ get newer alternatives or get reworked, some metrics may disappear or become obsolete. Like this one.

The Monitoring guide very explicitly states that there are different categories of metrics and that RabbitMQ will not provide them all. Disk activity is an infrastructure-level metric you can monitoring outside of RabbitMQ.

Almost every metric is exposed to Prometheus, and there are Grafana dashboards for more advanced metrics such as Raft and inter-node connection ones.