You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/reference/observability-metrics-mqtt-broker.md
+34-30Lines changed: 34 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,31 +20,35 @@ To configure options for these metrics, see [Configure MQTT broker diagnostic se
20
20
21
21
## MQTT Connect `metriccategory` user property
22
22
23
-
When a client connects to the broker, it can include a user property called `metriccategory` in the connect packet. The broker then tags all session-driven metrics (like publishes and subscribes) with this `metriccategory` as "category".
23
+
When a client connects to the broker, it can include a user property called `metriccategory` in the connect packet. The broker then tags all session-driven metrics (like publishes and subscribes) with this `metriccategory` as "category."
24
24
25
25
For example, if the self-check probe connects with `metriccategory=broker_selftest`, the broker tags all metrics from these sessions with "category=broker_selftest".
26
26
27
-
This helps dashboards show traffic sources without the high cardinality issues of tagging metrics with topics.
27
+
This feature helps dashboards show traffic sources without the high cardinality issues of tagging metrics with topics.
28
28
29
29
Sessions without a `metriccategory` are tagged as "category=uncategorized".
30
30
31
31
## Messaging metrics
32
32
33
-
All metrics contain the `hostname` tag, which distinguishes which pod the metrics were generated by. The tags listed in the individual metric descriptions below are in addition to this tag.
33
+
All metrics include the `hostname` tag to distinguish the pod that generated the metrics. The tags listed in the individual metric descriptions are additional to this tag.
34
+
35
+
## Messaging metrics
36
+
37
+
All metrics include the `hostname` tag to distinguish the pod that generated the metrics. The tags listed in the individual metric descriptions are additional to this tag.
34
38
35
39
| Metric | Description | Tags |
36
40
|--------|-------------|------|
37
-
|`aio_broker_publishes_received`| On the frontend, this represents how many incoming publish packets have been received from clients. For the backend, this represents how many internal messages have been sent from the frontend nodes. |`category`|
38
-
|`aio_broker_publishes_sent`| On the frontend, this represents how many outgoing publish packets have been sent to clients. If multiple clients are subscribed to the same topic, this will count each message sent, even if they have the same payload. This does not count ack packets. For the backend, this represents how many internal messages have been sent to the frontend nodes. |`category`|
39
-
|`aio_broker_payload_bytes_received`| The sum of the payloads of all publishes received. This does not include the size of the properties or publish packets themselves. |`category`|
40
-
|`aio_broker_payload_bytes_sent`| The sum of the payloads of all publishes sent. This does not include the size of the properties or publish packets themselves. |`category`|
41
-
|`aio_broker_authentication_successes`| This metric counts how many times a client has successfully authenticated. |`category`|
42
-
|`aio_broker_authentication_failures`| This metric counts how many times a client has failed to authenticate. Note for an errorless authentication server: aio_broker_authentication_successes + aio_broker_authentication_failures = aio_broker_publishes_received = publishes_sent |`category`|
43
-
|`aio_broker_authentication_deny`| This metric counts how many times a client was denied to authenticate. |`category`|
44
-
|`aio_broker_authorization_allow`| This metric counts how many times an authenticated client has successfully authorized. This should always be less than or equal to aio_broker_authentication_successes. |`category`|
45
-
|`aio_broker_authorization_deny`| This metric counts how many times an authenticated client has been denied. This should always be less than or equal to aio_broker_authentication_successes. |`category`|
46
-
|`aio_broker_qos0_messages_dropped`| This metric counts how many times a qos0 message was dropped for any reason. `direction`will be either `incoming` or `outgoing`. |`category`, `direction`|
47
-
|`aio_broker_backpressure_packets_rejected`| This metric counts how many times a packet has been rejected due to backpressure. A packet will be rejected if the system is at 97% capacity. ||
41
+
|`aio_broker_publishes_received`| On the frontend, this metric represents how many incoming publish packets are received from clients. For the backend, this metric represents how many internal messages are sent from the frontend nodes. |`category`|
42
+
|`aio_broker_publishes_sent`| On the frontend, this metric represents how many outgoing publish packets are sent to clients. If multiple clients are subscribed to the same topic, this metric counts each message sent, even if they have the same payload. This count doesn't count ack packets. For the backend, this metric represents how many internal messages are sent to the frontend nodes. |`category`|
43
+
|`aio_broker_payload_bytes_received`| The sum of the payloads of all publishes received. This sum doesn't include the size of the properties or publish packets themselves. |`category`|
44
+
|`aio_broker_payload_bytes_sent`| The sum of the payloads of all publishes sent. This sum doesn't include the size of the properties or publish packets themselves. |`category`|
45
+
|`aio_broker_authentication_successes`| This metric counts the number of successful authentication requests. |`category`|
46
+
|`aio_broker_authentication_failures`| This metric counts the number of failed authentication requests. For an errorless authentication server, `aio_broker_authentication_successes` + `aio_broker_authentication_failures` = `aio_broker_publishes_received = publishes_sent`|`category`|
47
+
|`aio_broker_authentication_deny`| This metric counts the number of denied authentication requests. |`category`|
48
+
|`aio_broker_authorization_allow`| This metric counts successfully authorization requests. This metric should always be less than or equal to `aio_broker_authentication_successes`. |`category`|
49
+
|`aio_broker_authorization_deny`| This metric counts denied authorization requests. This metric should always be less than or equal to `aio_broker_authentication_successes`. |`category`|
50
+
|`aio_broker_qos0_messages_dropped`| This metric counts the number of dropped QoS0 messages for any reason. The category `direction`is either `incoming` or `outgoing`. |`category`, `direction`|
51
+
|`aio_broker_backpressure_packets_rejected`| This metric counts number of rejected packets due to backpressure. A packet is rejected if the system is at 97% capacity. ||
48
52
|`aio_broker_store_retained_messages`| This metric counts how many retained messages are stored on the broker. ||
49
53
|`aio_broker_store_retained_bytes`| This metric counts how many bytes are stored via retained messages on the broker. ||
50
54
|`aio_broker_store_will_messages`| This metric counts how many will messages are stored on the broker. ||
@@ -54,7 +58,7 @@ All metrics contain the `hostname` tag, which distinguishes which pod the metric
54
58
55
59
This set of metrics tracks the [cardinality state of the broker](../manage-mqtt-broker/howto-configure-availability-scale.md). Each desired metric is paired with a reported metric to show the current state. These metrics indicate the number of healthy pods from the broker's perspective, which might differ from Kubernetes' reports.
56
60
57
-
For example, if a backend node restarts but hasn't reconnected to its chain, Kubernetes might report the pod as healthy, while the broker reports it as down because it's not functioning properly.
61
+
For example, if a backend node restarts but doesn't reconnect to its chain, there can be a discrepancy in health reports. Kubernetes might report the pod as healthy, while the broker reports it as down because it is not functioning properly.
58
62
59
63
| Desired Metric | Reported Metric |
60
64
|----------------|-----------------|
@@ -71,35 +75,35 @@ These metrics provide observability for the connections and subscriptions on the
71
75
72
76
| Metric | Description | Tags |
73
77
|--------|-------------|------|
74
-
|`aio_broker_total_sessions`| On the frontend and single node broker, this represents how many client sessions there are. This does not include disconnected persistent sessions, because a client might reconnect to a different frontend node. For the backend, this represents its connections to the other nodes in its chain. On the operator, this represents how many front and backend nodes are connected. For the authentication server, this represents how many frontend workers are connected (1 per frontend per thread). |`mqtt_version`: [v3/v5] <br> Backend Node Only Tags: <br> `is_tail`: [true/false] <br> `chain_id`: [n]|
75
-
|`aio_broker_store_total_sessions`| This is a backend specific metric that represents how many sessions are managed by the backend's chain. Backend nodes in the same chain should report the same number of sessions, and the sum of each chain should equal the sum of the frontend's total_sessions. |`is_persistent`: [true/false] <br> `is_tail`: [true/false] <br> `chain_id`: [n]|
78
+
|`aio_broker_total_sessions`| On the frontend and single node broker, this metric represents how many client sessions there are. This doesn't include disconnected persistent sessions, because a client might reconnect to a different frontend node. For the backend, this metric represents its connections to the other nodes in its chain. On the operator, this metric represents how many front and backend nodes are connected. For the authentication server, this metric represents how many frontend workers are connected (1 per frontend per thread). |`mqtt_version`: [v3/v5] <br> Backend Node Only Tags: <br> `is_tail`: [true/false] <br> `chain_id`: [n]|
79
+
|`aio_broker_store_total_sessions`| This metric represents how many sessions are in backend chain. Backend nodes in the same chain should report the same number of sessions, and the sum of each chain should equal the sum of the frontend's total_sessions. |`is_persistent`: [true/false] <br> `is_tail`: [true/false] <br> `chain_id`: [n]|
76
80
|`aio_broker_connected_sessions`| Same as `aio_broker_total_sessions`, except only sessions that have an active connection. ||
77
81
|`aio_broker_store_connected_sessions`| Same as `aio_broker_store_total_sessions`, except only sessions that have an active connection. If is_persistent is false, this should be equal to total sessions. ||
78
-
|`aio_broker_total_subscriptions`| On the frontend, this represents how many subscriptions the currently connected sessions have. This does not include disconnected persistent sessions, because a client might reconnect to a different frontend node. On the operator, this represents the frontend and backend nodes. For the authentication server, this represents how many frontend workers are connected (1 per frontend per thread). ||
79
-
|`aio_broker_store_total_subscriptions`| This is a backend specific metric that represents how many subscriptions are managed by the backend's chain. Backend nodes in the same chain should report the same number of subscriptions. This will not necessarily match the frontend's total_subscriptions, since this metric tracks disconnected persistent sessions as well. ||
82
+
|`aio_broker_total_subscriptions`| On the frontend, this metric represents how many subscriptions the currently connected sessions have. This doesn't include disconnected persistent sessions, because a client might reconnect to a different frontend node. On the operator, this metric represents the frontend and backend nodes. For the authentication server, this metric represents how many frontend workers are connected (1 per frontend per thread). ||
83
+
|`aio_broker_store_total_subscriptions`| This metric represents how many subscriptions are in backend chain. Backend nodes in the same chain should report the same number of subscriptions. This will not necessarily match the frontend's total_subscriptions, since this metric tracks disconnected persistent sessions as well. ||
80
84
81
85
## State store metrics
82
86
83
87
This set of metrics tracks the overall state of the [state store](../create-edge-apps/overview-state-store.md).
84
88
85
89
| Metric | Description | Tags |
86
90
|--------|-------------|------|
87
-
|`aio_broker_state_store_deletions`| This is a backend specific metric that represents how many times a request to delete a key has been received. This metric counts both successful deletes and errors. ||
88
-
|`aio_broker_state_store_insertions`| This is a backend specific metric that represents how many times a request to insert a new key has been received. This metric counts both successful insertions and errors. ||
89
-
|`aio_broker_state_store_keynotify_requests`| This is a backend specific metric that represents how many times a request to monitor keys' changing (KEYNOTIFY) has been received. This metric counts both successful modifications and errors. ||
90
-
|`aio_broker_state_store_modifications`| This is a backend specific metric that represents how many times a request to modify an existing key has been received. This metric counts both successful modifications and errors. ||
91
-
|`aio_broker_state_store_notifications_sent`| This is a backend specific metric that represents how many times the State Store sent a notification message in response to a key's value changing and a client registered via KEYNOTIFY. ||
92
-
|`aio_broker_state_store_retrievals`| This is a backend specific metric that represents how many times a request to request a key's value has been received. This metric counts both successful retrievals and errors. ||
91
+
|`aio_broker_state_store_deletions`| This metric counts the number of delete key requests received, including both successful deletes and errors. ||
92
+
|`aio_broker_state_store_insertions`| This metric counts the number of new key insert requests received, including both successful insertions and errors. ||
93
+
|`aio_broker_state_store_keynotify_requests`| This metric counts the number of requests to monitor key changes (KEYNOTIFY) received, including both successful modifications and errors. ||
94
+
|`aio_broker_state_store_modifications`| This metric counts the number of modify key requests received, including both successful modifications and errors. ||
95
+
|`aio_broker_state_store_notifications_sent`| This metric counts the number of notification messages sent by the state store in response to key value changes when a client is registered via KEYNOTIFY. ||
96
+
|`aio_broker_state_store_retrievals`| This metric counts the number of key value retrieval requests received, including both successful retrievals and errors. ||
93
97
94
98
## Disk-backed message buffer metrics
95
99
96
100
These metrics provide observability for the [disk-backed message buffer](../manage-mqtt-broker/howto-disk-backed-message-buffer.md).
97
101
98
102
| Metric | Description | Tags |
99
103
|--------|-------------|------|
100
-
|`aio_broker_buffer_pool_used_percent`|This metric reports the percentage of used buffer for a single frontend or backend buffer pool. |`name`|
101
-
|`aio_broker_disk_transfers_completed`|This metric reports the number of disk transfers completed on a given backend pod. It tracks the total number of publishes transferred from a buffer pool to disk. ||
102
-
|`aio_broker_disk_transfers_failed`|This metric reports the number of disk transfers that failed on a given backend pod. ||
104
+
|`aio_broker_buffer_pool_used_percent`|Reports the percentage of used buffer for a single frontend or backend buffer pool. |`name`|
105
+
|`aio_broker_disk_transfers_completed`|Reports the number of disk transfers completed on a given backend pod. Tracks the total number of publishes transferred from a buffer pool to disk. ||
106
+
|`aio_broker_disk_transfers_failed`|Reports the number of disk transfers that failed on a given backend pod. ||
103
107
104
108
> [!NOTE]
105
109
> Only certain backend buffer pools, specifically the dynamic ones named "reader", use the disk-backed message buffer feature. These pools store subscriber message queues and transfer elements to disk when usage exceeds 75%.
@@ -108,10 +112,10 @@ These metrics provide observability for the [disk-backed message buffer](../mana
108
112
109
113
| Metric | Description |
110
114
|--------|-------------|
111
-
|`aio_broker_store_transfer_batch_receiver_message_count`| This metric reports the number of messages received by the store transfer receiver. This should be equal to the number of messages sent by the store transfer sender. |
115
+
|`aio_broker_store_transfer_batch_receiver_message_count`| This metric reports the number of messages received by the store transfer receiver. This count should be equal to the number of messages sent by the store transfer sender. |
112
116
|`aio_broker_store_transfer_batch_sender_transfer_bytes`| This metric reports the number of bytes sent by the store transfer sender. |
113
117
|`aio_broker_store_transfer_batch_sender_message_count`| This metric reports the number of messages sent by the store transfer sender. |
114
-
|`aio_broker_store_transfer_ack_event_receiver_message_count`| This metric reports the number of ack event messages received by the store transfer receiver. This should be equal to the number of ack event messages sent by the store transfer sender. |
118
+
|`aio_broker_store_transfer_ack_event_receiver_message_count`| This metric reports the number of ack event messages received by the store transfer receiver. This count should be equal to the number of ack event messages sent by the store transfer sender. |
115
119
|`aio_broker_store_transfer_ack_event_sender_message_count`| This metric reports the number of ack event messages sent by the store transfer sender. |
116
120
|`aio_broker_store_transfer_ack_event_sender_transfer_bytes`| This metric reports the number of bytes sent by the store transfer sender for ack events. |
117
121
|`aio_broker_store_transfer_patch_tracker_receiver_message_count`| This metric reports the number of patch tracker messages received by the store transfer receiver. This should be equal to the number of patch tracker messages sent by the store transfer sender. |
0 commit comments