Skip to content

Commit 675b6b8

Browse files
authored
Added information about trial and production instance limits (#518)
* Added information about trial and production instance limits * Updated sentence before lists in line with S&G * Removed linebreak.
1 parent 4c4aa08 commit 675b6b8

File tree

1 file changed

+52
-8
lines changed

1 file changed

+52
-8
lines changed

docs/kafka/metrics-monitoring-kafka/README.adoc

Lines changed: 52 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -86,27 +86,71 @@ As a developer or administrator, you can view metrics in {product-kafka} to visu
8686
Cluster metrics::
8787
+
8888
--
89-
* `kafka_namespace:haproxy_server_bytes_in_total:rate5m`: Number of incoming bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are sending to topics in the cluster. The maximum incoming byte rate is 30 MB per second.
89+
* `kafka_namespace:haproxy_server_bytes_in_total:rate5m`: Number of incoming bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are sending to topics in the cluster.
90+
+
91+
The Kafka instance type determines the maximum incoming byte rate, as shown in the following values:
92+
93+
** 1 MiB per second for a trial instance
94+
** 50 MiB per second for a production instance with a size of 1 streaming unit
95+
** 100 MiB per second for a production instance with a size of 2 streaming units
96+
97+
* `kafka_namespace:haproxy_server_bytes_out_total:rate5m`: Number of outgoing bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are receiving from topics in the cluster.
98+
+
99+
The Kafka instance type determines the maximum outgoing byte rate, as shown in the following values:
100+
101+
** 1 MiB per second for a trial instance
102+
** 100 MiB per second for a production instance with a size of 1 streaming unit
103+
** 200 MiB per second for a production instance with a size of 2 streaming units
104+
105+
* `kafka_namespace:kafka_server_socket_server_metrics_connection_count:sum`: Number of current client connections to the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. For example, a consumer holds a connection to each broker it is receiving data from and a connection to its group coordinator.
106+
+
107+
The Kafka instance type determines the maximum number of active connections, as shown in the following values:
108+
109+
** 100 for a trial instance
110+
** 3000 for a production instance with a size of 1 streaming unit
111+
** 6000 for a production instance with a size of 2 streaming units
90112

91-
* `kafka_namespace:haproxy_server_bytes_out_total:rate5m`: Number of outgoing bytes per second for the cluster in the last five minutes. This metric represents all the data that producers are receiving from topics in the cluster. The maximum outgoing byte rate is 30 MB per second.
113+
* `kafka_namespace:kafka_server_socket_server_metrics_connection_creation_rate:sum`: Number of client connection creations per second for the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. A constant high number of connection creations might indicate a client issue.
114+
+
115+
The Kafka instance type determines the maximum connection creation rate, as shown in the following values:
92116

93-
* `kafka_namespace:kafka_server_socket_server_metrics_connection_count:sum`: Number of current client connections to the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. For example, a consumer holds a connection to each broker it is receiving data from and a connection to its group coordinator. The maximum number of active connections is 2000.
117+
** 50 per second for a trial instance
118+
** 100 per second for a production instance with a size of 1 streaming unit
119+
** 200 per second for a production instance with a size of 2 streaming units
94120

95-
* `kafka_namespace:kafka_server_socket_server_metrics_connection_creation_rate:sum`: Number of client connection creations per second for the cluster. Kafka clients use persistent connections to interact with brokers in the cluster. A constant high number of connection creations might indicate a client issue. The maximum connection creation rate is 100 per second.
121+
* `kafka_topic:kafka_topic_partitions:count`: Number of topics in the cluster. This does not include internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of topics is 1000 for all Kafka instances.
96122

97-
* `kafka_topic:kafka_topic_partitions:count`: Number of topics in the cluster. This does not include internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of topics is 1000.
123+
* `kafka_topic:kafka_topic_partitions:sum`: Number of partitions across all topics in the cluster. This does not include partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`.
124+
+
125+
The Kafka instance type determines the maximum number of partitions, as shown in the following values:
98126

99-
* `kafka_topic:kafka_topic_partitions:sum`: Number of partitions across all topics in the cluster. This does not include partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of partitions is 1000.
127+
** 100 for a trial instance
128+
** 1500 for a production instance with a size of 1 streaming unit
129+
** 3000 for a production instance with a size of 2 streaming units
100130
--
101131

102132
Broker metrics::
103133
+
104134
--
105-
* `kafka_broker_quota_softlimitbytes`: Maximum amount of storage, in bytes, for this broker before producers are throttled. When this limit is reached, the broker starts throttling producers to prevent them from sending additional data. The maximum storage in the broker is 1000 GiB.
135+
* `kafka_broker_quota_softlimitbytes`: Maximum amount of storage, in bytes, for this broker before producers are throttled. When this limit is reached, the broker starts throttling producers to prevent them from sending additional data.
136+
+
137+
The Kafka instance type determines the maximum storage in the broker, as shown in the following values:
138+
139+
** 10 GiB for a trial instance
140+
** 1000 GiB for a production instance with a size of 1 streaming unit
141+
** 2000 GiB for a production instance with a size of 2 streaming units
142+
106143

107144
* `kafka_broker_quota_totalstorageusedbytes`: Amount of storage, in bytes, that is currently used by partitions in the broker. The storage usage depends on the number and retention configurations of the partitions. This metric must stay below the `kafka_broker_quota_softlimitbytes` metric setting.
108145

109-
* `kafka_controller_kafkacontroller_global_partition_count`: Number of partitions in the cluster. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`. This count includes partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. The maximum number of partitions is 1000. This metric is similar to the `kafka_topic:kafka_topic_partitions:sum` cluster metric.
146+
* `kafka_controller_kafkacontroller_global_partition_count`: Number of partitions in the cluster. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`. This count includes partitions from internal Kafka topics, such as `\__consumer_offsets` and `__transaction_state`. This metric is similar to the `kafka_topic:kafka_topic_partitions:sum` cluster metric.
147+
+
148+
The Kafka instance type determines the maximum storage in the broker, as shown in the following values:
149+
150+
** 100 for a trial instance
151+
** 1500 for a production instance with a size of 1 streaming unit
152+
** 3000 for a production instance with a size of 2 streaming units +
153+
110154

111155
* `kafka_controller_kafkacontroller_offline_partitions_count`: Number of partitions in the cluster that are currently offline. Offline partitions cannot be used by clients for producing or consuming data. Only the broker that is the current controller in the cluster reports this metric. Any other brokers report `0`.
112156

0 commit comments

Comments
 (0)