Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,33 @@

Optimize the availability of your clusters by configuring and tuning properties.

// Don't display ListKafkaConnections in Cloud docs until support is added
ifdef::env-cloud[]
TIP: Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections. Use the xref:reference:public-metrics-reference.adoc#redpanda_rpc_active_connections[`redpanda_rpc_active_connections` metric] to view current Kafka client connections.
[TIP]
====
Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections.

* Use the xref:reference:public-metrics-reference.adoc#redpanda_rpc_active_connections[`redpanda_rpc_active_connections` metric] to view current Kafka client connections.
* For clusters on v25.3 and later, use xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] or the `GET /v1/monitoring/kafka/connections` endpoint in the Data Plane API to identify:
+
--
** Which clients and applications are connected
** Long-lived connections and long-running requests
** Connections with no activity
** Whether any clients are causing excessive load
--
+
By reviewing connection details, you can make informed decisions about tuning connection limits and troubleshooting issues.

See also: link:/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections[Data Plane API reference], xref:manage:monitor-cloud.adoc#throughput[Monitor Redpanda Cloud]
====
endif::[]
ifndef::env-cloud[]
[TIP]
====
Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections.

* Internal metrics that follow the `vectorized_kafka_rpc_.\*connect.*` naming pattern provide details on Kafka client connection activity. For example, xref:reference:internal-metrics-reference.adoc#vectorized_kafka_rpc_active_connections[`vectorized_kafka_rpc_active_connections`] reports the current number of active connections.
* For Redpanda v25.3 and later, use `rpk cluster connections list` or the Admin API ListKafkaConnections endpoint to identify:
* Internal metrics that follow the `vectorized_kafka_rpc_.\*connect*` naming pattern provide details on Kafka client connection activity. For example, xref:reference:internal-metrics-reference.adoc#vectorized_kafka_rpc_active_connections[`vectorized_kafka_rpc_active_connections`] reports the current number of active connections.
* For Redpanda v25.3 and later, use xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] or the Admin API ListKafkaConnections endpoint to identify:
+
--
** Which clients and applications are connected
Expand Down
193 changes: 189 additions & 4 deletions modules/manage/pages/cluster-maintenance/manage-throughput.adoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
= Manage Throughput
:description: Learn how to manage the throughput of Kafka traffic.
:page-categories: Management, Networking

// tag::single-source[]
ifdef::env-cloud[:monitor-doc: manage:monitor-cloud.adoc#throughput]
ifndef::env-cloud[:monitor-doc: manage:monitoring.adoc#throughput]
ifdef::env-cloud[]
:monitor-doc: manage:monitor-cloud.adoc#throughput
:connected-clients-api-doc-ref: link:/api/doc/cloud-dataplane/operation/operation-monitoringservice_listkafkaconnections
endif::[]
ifndef::env-cloud[]
:monitor-doc: manage:monitoring.adoc#throughput
:connected-clients-api-doc-ref: link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections
endif::[]

Redpanda supports throughput throttling on both ingress and egress independently, and allows configuration at the broker and client levels. This helps prevent clients from causing unbounded network and disk usage on brokers. You can configure limits at two levels:

Expand All @@ -21,7 +28,13 @@ endif::[]

You may find it helpful to check the xref:{monitor-doc}[current produce and consume throughput] of a client before you configure throughput quotas.

Use the `rpk cluster connections list` command or the link:/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections[ListKafkaConnections] Admin API endpoint to view detailed information about active Kafka client connections.
ifndef::env-cloud[]
Use the xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] command or the {connected-clients-api-doc-ref}[ListKafkaConnections] Admin API endpoint to view detailed information about active Kafka client connections.
endif::[]

ifdef::env-cloud[]
Use the xref:reference:rpk/rpk-cluster/rpk-cluster-connections-list.adoc[`rpk cluster connections list`] command or the {connected-clients-api-doc-ref}[`GET /v1/monitoring/kafka/connections`] Data Plane API endpoint to view detailed information about active Kafka client connections.
endif::[]

For example, to view a cluster's connected clients in order of highest current produce throughput, run:

Expand Down Expand Up @@ -50,6 +63,7 @@ b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-clie
----
--

ifndef::env-cloud[]
curl::
+
--
Expand All @@ -67,7 +81,7 @@ curl -s -X POST \
.Show example API response
[%collapsible]
====
[,json,role=no-copy,lines=54]
[,json,role=no-copy,lines=55]
----
{
"connections": [
Expand Down Expand Up @@ -135,6 +149,95 @@ curl -s -X POST \
----
====
--
endif::[]

ifdef::env-cloud[]
Data Plane API::
+
--
[,bash]
----
curl \
--request GET 'https://<dataplane-api-url>/v1/monitoring/kafka/connections' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '{
"filter": "",
"order_by": "recent_request_statistics.produce_bytes desc"
}'
----

.Show example API response
[%collapsible]
====
[,json,role=no-copy,lines=54]
----
{
"connections": [
{
"node_id": 0,
"shard_id": 0,
"uid": "b20601a3-624c-4a8c-ab88-717643f01d56",
"state": "KAFKA_CONNECTION_STATE_OPEN",
"open_time": "2025-10-15T14:15:15.755065000Z",
"close_time": "1970-01-01T00:00:00.000000000Z",
"authentication_info": {
"state": "AUTHENTICATION_STATE_UNAUTHENTICATED",
"mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED",
"user_principal": ""
},
"listener_name": "",
"tls_info": {
"enabled": false
},
"source": {
"ip_address": "127.0.0.1",
"port": 55012
},
"client_id": "perf-producer-client",
"client_software_name": "apache-kafka-java",
"client_software_version": "3.9.0",
"transactional_id": "my-tx-id",
"group_id": "",
"group_instance_id": "",
"group_member_id": "",
"api_versions": {
"18": 4,
"22": 3,
"3": 12,
"24": 3,
"0": 7
},
"idle_duration": "0s",
"in_flight_requests": {
"sampled_in_flight_requests": [
{
"api_key": 0,
"in_flight_duration": "0.000406892s"
}
],
"has_more_requests": false
},
"total_request_statistics": {
"produce_bytes": "78927173",
"fetch_bytes": "0",
"request_count": "4853",
"produce_batch_count": "4849"
},
"recent_request_statistics": {
"produce_bytes": "78927173",
"fetch_bytes": "0",
"request_count": "4853",
"produce_batch_count": "4849"
}
},
...
],
"total_size": "9"
}
----
====
--
endif::[]
======

To view connections for a specific client, you can use a filter expression:
Expand All @@ -157,6 +260,7 @@ b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-clie
----
--

ifndef::env-cloud[]
curl::
+
--
Expand Down Expand Up @@ -234,6 +338,87 @@ curl -s -X POST \
----
====
--
endif::[]

ifdef::env-cloud[]
Data Plane API::
+
--
[,bash]
----
curl \
--request GET 'https://<dataplane-api-url>/v1/monitoring/kafka/connections' \
--header "Authorization: Bearer $ACCESS_TOKEN" \
--data '{
"filter": "client_id = \"perf-producer-client\""
}'
----

.Show example API response
[%collapsible]
====
[,json,lines=23]
----
{
"connections": [
{
"node_id": 0,
"shard_id": 0,
"uid": "b41584f3-2662-4185-a4b8-0d8510f5c780",
"state": "KAFKA_CONNECTION_STATE_OPEN",
"open_time": "2025-10-15T14:15:15.219538000Z",
"close_time": "1970-01-01T00:00:00.000000000Z",
"authentication_info": {
"state": "AUTHENTICATION_STATE_UNAUTHENTICATED",
"mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED",
"user_principal": ""
},
"listener_name": "",
"tls_info": {
"enabled": false
},
"source": {
"ip_address": "127.0.0.1",
"port": 55002
},
"client_id": "perf-producer-client",
"client_software_name": "apache-kafka-java",
"client_software_version": "3.9.0",
"transactional_id": "",
"group_id": "",
"group_instance_id": "",
"group_member_id": "",
"api_versions": {
"18": 4,
"3": 12,
"10": 4
},
"idle_duration": "7.743592270s",
"in_flight_requests": {
"sampled_in_flight_requests": [],
"has_more_requests": false
},
"total_request_statistics": {
"produce_bytes": "0",
"fetch_bytes": "0",
"request_count": "3",
"produce_batch_count": "0"
},
"recent_request_statistics": {
"produce_bytes": "0",
"fetch_bytes": "0",
"request_count": "3",
"produce_batch_count": "0"
}
},
...
],
"total_size": "2"
}
----
====
--
endif::[]
======


Expand Down
Loading