You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/kafka/metrics-monitoring-kafka/README.adoc
+52-8Lines changed: 52 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -86,27 +86,71 @@ As a developer or administrator, you can view metrics in {product-kafka} to visu
86
86
Cluster metrics::
87
87
+
88
88
--
89
-
* `kafka_namespace:haproxy_server_bytes_in_total:rate5m`: Number of incoming bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are sending to topics in the cluster. The maximum incoming byte rate is 30 MB per second.
89
+
* `kafka_namespace:haproxy_server_bytes_in_total:rate5m`: Number of incoming bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are sending to topics in the cluster.
90
+
+
91
+
The Kafka instance type determines the maximum incoming byte rate, as shown in the following values:
92
+
93
+
** 1 MiB per second for a trial instance
94
+
** 50 MiB per second for a production instance with a size of 1 streaming unit
95
+
** 100 MiB per second for a production instance with a size of 2 streaming units
96
+
97
+
* `kafka_namespace:haproxy_server_bytes_out_total:rate5m`: Number of outgoing bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are receiving from topics in the cluster.
98
+
+
99
+
The Kafka instance type determines the maximum outgoing byte rate, as shown in the following values:
100
+
101
+
** 1 MiB per second for a trial instance
102
+
** 100 MiB per second for a production instance with a size of 1 streaming unit
103
+
** 200 MiB per second for a production instance with a size of 2 streaming units
104
+
105
+
* `kafka_namespace:kafka_server_socket_server_metrics_connection_count:sum`: Number of current client connections to the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. For example, a consumer holds a connection to each broker it is receiving data from and a connection to its group coordinator.
106
+
+
107
+
The Kafka instance type determines the maximum number of active connections, as shown in the following values:
108
+
109
+
** 100 for a trial instance
110
+
** 3000 for a production instance with a size of 1 streaming unit
111
+
** 6000 for a production instance with a size of 2 streaming units
90
112
91
-
* `kafka_namespace:haproxy_server_bytes_out_total:rate5m`: Number of outgoing bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are receiving from topics in the cluster. The maximum outgoing byte rate is 30 MB per second.
113
+
* `kafka_namespace:kafka_server_socket_server_metrics_connection_creation_rate:sum`: Number of client connection creations per second for the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. A constant high number of connection creations might indicate a client issue.
114
+
+
115
+
The Kafka instance type determines the maximum connection creation rate, as shown in the following values:
92
116
93
-
* `kafka_namespace:kafka_server_socket_server_metrics_connection_count:sum`: Number of current client connections to the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. For example, a consumer holds a connection to each broker it is receiving data from and a connection to its group coordinator. The maximum number of active connections is 2000.
117
+
** 50 per second for a trial instance
118
+
** 100 per second for a production instance with a size of 1 streaming unit
119
+
** 200 per second for a production instance with a size of 2 streaming units
94
120
95
-
* `kafka_namespace:kafka_server_socket_server_metrics_connection_creation_rate:sum`: Number of client connection creations per second for the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. A constant high number of connection creations might indicate a client issue. The maximum connection creation rate is 100 per second.
121
+
* `kafka_topic:kafka_topic_partitions:count`: Number of topics in the cluster. This does not include internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of topics is 1000 for all Kafka instances.
96
122
97
-
* `kafka_topic:kafka_topic_partitions:count`: Number of topics in the cluster. This does not include internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of topics is 1000.
123
+
* `kafka_topic:kafka_topic_partitions:sum`: Number of partitions across all topics in the cluster. This does not include partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`.
124
+
+
125
+
The Kafka instance type determines the maximum number of partitions, as shown in the following values:
98
126
99
-
* `kafka_topic:kafka_topic_partitions:sum`: Number of partitions across all topics in the cluster. This does not include partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of partitions is 1000.
127
+
** 100 for a trial instance
128
+
** 1500 for a production instance with a size of 1 streaming unit
129
+
** 3000 for a production instance with a size of 2 streaming units
100
130
--
101
131
102
132
Broker metrics::
103
133
+
104
134
--
105
-
* `kafka_broker_quota_softlimitbytes`: Maximum amount of storage, in bytes, for this broker before producers are throttled. When this limit is reached, the broker starts throttling producers to prevent them from sending additional data. The maximum storage in the broker is 1000 GiB.
135
+
* `kafka_broker_quota_softlimitbytes`: Maximum amount of storage, in bytes, for this broker before producers are throttled. When this limit is reached, the broker starts throttling producers to prevent them from sending additional data.
136
+
+
137
+
The Kafka instance type determines the maximum storage in the broker, as shown in the following values:
138
+
139
+
** 10 GiB for a trial instance
140
+
** 1000 GiB for a production instance with a size of 1 streaming unit
141
+
** 2000 GiB for a production instance with a size of 2 streaming units
142
+
106
143
107
144
* `kafka_broker_quota_totalstorageusedbytes`: Amount of storage, in bytes, that is currently used by partitions in the broker. The storage usage depends on the number and retention configurations of the partitions. This metric must stay below the `kafka_broker_quota_softlimitbytes` metric setting.
108
145
109
-
* `kafka_controller_kafkacontroller_global_partition_count`: Number of partitions in the cluster. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`. This count includes partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of partitions is 1000. This metric is similar to the `kafka_topic:kafka_topic_partitions:sum` cluster metric.
146
+
* `kafka_controller_kafkacontroller_global_partition_count`: Number of partitions in the cluster. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`. This count includes partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. This metric is similar to the `kafka_topic:kafka_topic_partitions:sum` cluster metric.
147
+
+
148
+
The Kafka instance type determines the maximum storage in the broker, as shown in the following values:
149
+
150
+
** 100 for a trial instance
151
+
** 1500 for a production instance with a size of 1 streaming unit
152
+
** 3000 for a production instance with a size of 2 streaming units +
153
+
110
154
111
155
* `kafka_controller_kafkacontroller_offline_partitions_count`: Number of partitions in the cluster that are currently offline. Offline partitions cannot be used by clients for producing or consuming data. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`.
0 commit comments