Skip to content

Commit bd2b8ff

Browse files
kbatuigasFeediver1
andauthored
Single source connected client monitoring docs for Cloud (#1470)
Co-authored-by: Joyce Fee <[email protected]>
1 parent df3d231 commit bd2b8ff

File tree

5 files changed

+325
-17
lines changed

5 files changed

+325
-17
lines changed

modules/manage/pages/cluster-maintenance/configure-client-connections.adoc

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,33 @@
66

77
Optimize the availability of your clusters by configuring and tuning properties.
88

9-
// Don't display ListKafkaConnections in Cloud docs until support is added
109
ifdef::env-cloud[]
11-
TIP: Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections. Use the xref:reference:public-metrics-reference.adoc#redpanda_rpc_active_connections[`redpanda_rpc_active_connections` metric] to view current Kafka client connections.
10+
[TIP]
11+
====
12+
Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections.
13+
14+
* Use the xref:reference:public-metrics-reference.adoc#redpanda_rpc_active_connections[`redpanda_rpc_active_connections` metric] to view current Kafka client connections.
15+
* For clusters on v25.3 and later, use xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] or the `GET /v1/monitoring/kafka/connections` endpoint in the Data Plane API to identify:
16+
+
17+
--
18+
** Which clients and applications are connected
19+
** Long-lived connections and long-running requests
20+
** Connections with no activity
21+
** Whether any clients are causing excessive load
22+
--
23+
+
24+
By reviewing connection details, you can make informed decisions about tuning connection limits and troubleshooting issues.
25+
26+
See also: link:/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections[Data Plane API reference], xref:manage:monitor-cloud.adoc#throughput[Monitor Redpanda Cloud]
27+
====
1228
endif::[]
1329
ifndef::env-cloud[]
1430
[TIP]
1531
====
1632
Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections.
1733

18-
* Internal metrics that follow the `vectorized_kafka_rpc_.\*connect.*` naming pattern provide details on Kafka client connection activity. For example, xref:reference:internal-metrics-reference.adoc#vectorized_kafka_rpc_active_connections[`vectorized_kafka_rpc_active_connections`] reports the current number of active connections.
19-
* For Redpanda v25.3 and later, use `rpk cluster connections list` or the Admin API ListKafkaConnections endpoint to identify:
34+
* Internal metrics that follow the `vectorized_kafka_rpc_.\*connect*` naming pattern provide details on Kafka client connection activity. For example, xref:reference:internal-metrics-reference.adoc#vectorized_kafka_rpc_active_connections[`vectorized_kafka_rpc_active_connections`] reports the current number of active connections.
35+
* For Redpanda v25.3 and later, use xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] or the Admin API ListKafkaConnections endpoint to identify:
2036
+
2137
--
2238
** Which clients and applications are connected

modules/manage/pages/cluster-maintenance/manage-throughput.adoc

Lines changed: 189 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
11
= Manage Throughput
22
:description: Learn how to manage the throughput of Kafka traffic.
33
:page-categories: Management, Networking
4+
45
// tag::single-source[]
5-
ifdef::env-cloud[:monitor-doc: manage:monitor-cloud.adoc#throughput]
6-
ifndef::env-cloud[:monitor-doc: manage:monitoring.adoc#throughput]
6+
ifdef::env-cloud[]
7+
:monitor-doc: manage:monitor-cloud.adoc#throughput
8+
:connected-clients-api-doc-ref: link:/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections
9+
endif::[]
10+
ifndef::env-cloud[]
11+
:monitor-doc: manage:monitoring.adoc#throughput
12+
:connected-clients-api-doc-ref: link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections
13+
endif::[]
714

815
Redpanda supports throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. This helps prevent clients from causing unbounded network and disk usage on brokers. You can configure limits at two levels:
916

@@ -21,7 +28,13 @@ endif::[]
2128

2229
You may find it helpful to check the xref:{monitor-doc}[current produce and consume throughput] of a client before you configure throughput quotas.
2330

24-
Use the `rpk cluster connections list` command or the link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections[ListKafkaConnections] Admin API endpoint to view detailed information about active Kafka client connections.
31+
ifndef::env-cloud[]
32+
Use the xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] command or the {connected-clients-api-doc-ref}[ListKafkaConnections] Admin API endpoint to view detailed information about active Kafka client connections.
33+
endif::[]
34+
35+
ifdef::env-cloud[]
36+
Use the xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] command or the {connected-clients-api-doc-ref}[`GET /v1/monitoring/kafka/connections`] Data Plane API endpoint to view detailed information about active Kafka client connections.
37+
endif::[]
2538

2639
For example, to view a cluster's connected clients in order of highest current produce throughput, run:
2740

@@ -50,6 +63,7 @@ b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-clie
5063
----
5164
--
5265
66+
ifndef::env-cloud[]
5367
curl::
5468
+
5569
--
@@ -67,7 +81,7 @@ curl -s -X POST \
6781
.Show example API response
6882
[%collapsible]
6983
====
70-
[,json,role=no-copy,lines=54]
84+
[,json,role=no-copy,lines=55]
7185
----
7286
{
7387
"connections": [
@@ -135,6 +149,95 @@ curl -s -X POST \
135149
----
136150
====
137151
--
152+
endif::[]
153+
154+
ifdef::env-cloud[]
155+
Data Plane API::
156+
+
157+
--
158+
[,bash]
159+
----
160+
curl \
161+
--request GET 'https://<dataplane-api-url>/v1/monitoring/kafka/connections' \
162+
--header "Authorization: Bearer $ACCESS_TOKEN" \
163+
--data '{
164+
"filter": "",
165+
"order_by": "recent_request_statistics.produce_bytes desc"
166+
}'
167+
----
168+
169+
.Show example API response
170+
[%collapsible]
171+
====
172+
[,json,role=no-copy,lines=54]
173+
----
174+
{
175+
"connections": [
176+
{
177+
"node_id": 0,
178+
"shard_id": 0,
179+
"uid": "b20601a3-624c-4a8c-ab88-717643f01d56",
180+
"state": "KAFKA_CONNECTION_STATE_OPEN",
181+
"open_time": "2025-10-15T14:15:15.755065000Z",
182+
"close_time": "1970-01-01T00:00:00.000000000Z",
183+
"authentication_info": {
184+
"state": "AUTHENTICATION_STATE_UNAUTHENTICATED",
185+
"mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED",
186+
"user_principal": ""
187+
},
188+
"listener_name": "",
189+
"tls_info": {
190+
"enabled": false
191+
},
192+
"source": {
193+
"ip_address": "127.0.0.1",
194+
"port": 55012
195+
},
196+
"client_id": "perf-producer-client",
197+
"client_software_name": "apache-kafka-java",
198+
"client_software_version": "3.9.0",
199+
"transactional_id": "my-tx-id",
200+
"group_id": "",
201+
"group_instance_id": "",
202+
"group_member_id": "",
203+
"api_versions": {
204+
"18": 4,
205+
"22": 3,
206+
"3": 12,
207+
"24": 3,
208+
"0": 7
209+
},
210+
"idle_duration": "0s",
211+
"in_flight_requests": {
212+
"sampled_in_flight_requests": [
213+
{
214+
"api_key": 0,
215+
"in_flight_duration": "0.000406892s"
216+
}
217+
],
218+
"has_more_requests": false
219+
},
220+
"total_request_statistics": {
221+
"produce_bytes": "78927173",
222+
"fetch_bytes": "0",
223+
"request_count": "4853",
224+
"produce_batch_count": "4849"
225+
},
226+
"recent_request_statistics": {
227+
"produce_bytes": "78927173",
228+
"fetch_bytes": "0",
229+
"request_count": "4853",
230+
"produce_batch_count": "4849"
231+
}
232+
},
233+
...
234+
],
235+
"total_size": "9"
236+
}
237+
----
238+
====
239+
--
240+
endif::[]
138241
======
139242

140243
To view connections for a specific client, you can use a filter expression:
@@ -157,6 +260,7 @@ b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-clie
157260
----
158261
--
159262
263+
ifndef::env-cloud[]
160264
curl::
161265
+
162266
--
@@ -234,6 +338,87 @@ curl -s -X POST \
234338
----
235339
====
236340
--
341+
endif::[]
342+
343+
ifdef::env-cloud[]
344+
Data Plane API::
345+
+
346+
--
347+
[,bash]
348+
----
349+
curl \
350+
--request GET 'https://<dataplane-api-url>/v1/monitoring/kafka/connections' \
351+
--header "Authorization: Bearer $ACCESS_TOKEN" \
352+
--data '{
353+
"filter": "client_id = \"perf-producer-client\""
354+
}'
355+
----
356+
357+
.Show example API response
358+
[%collapsible]
359+
====
360+
[,json,lines=23]
361+
----
362+
{
363+
"connections": [
364+
{
365+
"node_id": 0,
366+
"shard_id": 0,
367+
"uid": "b41584f3-2662-4185-a4b8-0d8510f5c780",
368+
"state": "KAFKA_CONNECTION_STATE_OPEN",
369+
"open_time": "2025-10-15T14:15:15.219538000Z",
370+
"close_time": "1970-01-01T00:00:00.000000000Z",
371+
"authentication_info": {
372+
"state": "AUTHENTICATION_STATE_UNAUTHENTICATED",
373+
"mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED",
374+
"user_principal": ""
375+
},
376+
"listener_name": "",
377+
"tls_info": {
378+
"enabled": false
379+
},
380+
"source": {
381+
"ip_address": "127.0.0.1",
382+
"port": 55002
383+
},
384+
"client_id": "perf-producer-client",
385+
"client_software_name": "apache-kafka-java",
386+
"client_software_version": "3.9.0",
387+
"transactional_id": "",
388+
"group_id": "",
389+
"group_instance_id": "",
390+
"group_member_id": "",
391+
"api_versions": {
392+
"18": 4,
393+
"3": 12,
394+
"10": 4
395+
},
396+
"idle_duration": "7.743592270s",
397+
"in_flight_requests": {
398+
"sampled_in_flight_requests": [],
399+
"has_more_requests": false
400+
},
401+
"total_request_statistics": {
402+
"produce_bytes": "0",
403+
"fetch_bytes": "0",
404+
"request_count": "3",
405+
"produce_batch_count": "0"
406+
},
407+
"recent_request_statistics": {
408+
"produce_bytes": "0",
409+
"fetch_bytes": "0",
410+
"request_count": "3",
411+
"produce_batch_count": "0"
412+
}
413+
},
414+
...
415+
],
416+
"total_size": "2"
417+
}
418+
----
419+
====
420+
--
421+
endif::[]
237422
======
238423

239424

0 commit comments

Comments
 (0)