|
1 | 1 | # Kafka Connect Metrics |
2 | 2 |
|
3 | 3 | Here is the list of metrics based on MBeans exposed by Apache Kafka Connect. String-valued JMX |
4 | | -attributes (class/type/version information) are exported as `UpDownCounter` metrics with value `1` |
5 | | -and carry the raw string value as metric attributes. |
| 4 | +attributes are exported as `UpDownCounter` metrics with value `1` and only include connector/task |
| 5 | +identifiers alongside any state-mapping attributes. |
6 | 6 |
|
7 | 7 | ## Compatibility |
8 | 8 |
|
9 | 9 | This rule set targets both Apache Kafka Connect and Confluent Platform. Apache documents several |
10 | 10 | metrics not surfaced in Confluent docs (worker rebalance protocol, per-connector task counts on |
11 | | -workers, predicate/transform metadata, connector task metadata including converter info, source |
12 | | -transaction size stats, and sink record lag max); all of them are included below. Status metrics use |
| 11 | +workers, source transaction size stats, and sink record lag max); all of them are included below. Status metrics use |
13 | 12 | the superset of values across both variants (connector: running, paused, stopped, failed, |
14 | 13 | restarting, unassigned, degraded; task: running, paused, failed, restarting, unassigned, |
15 | 14 | destroyed) and fall back to `unknown` for any new values. Differences in bean placeholder |
@@ -46,74 +45,41 @@ formatting between the docs are cosmetic; bean names align across both variants. |
46 | 45 |
|
47 | 46 | ## Worker rebalance metrics |
48 | 47 |
|
49 | | -All metrics include `kafka.connect.worker.leader`. |
50 | | - |
51 | 48 | | Metric Name | Type | Unit | Attributes | Description | |
52 | 49 | |------------------------------------------------|---------------|-------------|--------------------------------------|---------------------------------------------------------------------------------| |
53 | 50 | | kafka.connect.worker.rebalance.completed.total | Counter | {rebalance} | | The total number of rebalances completed by this worker. | |
54 | 51 | | kafka.connect.worker.rebalance.protocol | UpDownCounter | 1 | kafka.connect.protocol.state | The Connect protocol used by this cluster. | |
55 | 52 | | kafka.connect.worker.rebalance.epoch | UpDownCounter | {epoch} | | The epoch or generation number of this worker. | |
56 | | -| kafka.connect.worker.rebalance.leader | UpDownCounter | 1 | kafka.connect.worker.leader.state | The name of the group leader. | |
57 | 53 | | kafka.connect.worker.rebalance.avg.time | Gauge | s | | The average time in milliseconds spent by this worker to rebalance. | |
58 | 54 | | kafka.connect.worker.rebalance.max.time | Gauge | s | | The maximum time in milliseconds spent by this worker to rebalance. | |
59 | 55 | | kafka.connect.worker.rebalance.active | UpDownCounter | 1 | kafka.connect.worker.rebalance.state | Whether this worker is currently rebalancing. | |
60 | 56 | | kafka.connect.worker.rebalance.since_last | Gauge | s | | The time in milliseconds since this worker completed the most recent rebalance. | |
61 | 57 |
|
62 | 58 | ## Connector metrics |
63 | 59 |
|
64 | | -Attributes: `kafka.connect.connector`, `kafka.connect.connector.class`, `kafka.connect.connector.version`, `kafka.connect.connector.type.raw`. |
65 | | - |
66 | | -| Metric Name | Type | Unit | Attributes | Description | |
67 | | -|---------------------------------|---------------|------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
68 | | -| kafka.connect.connector.class | UpDownCounter | 1 | kafka.connect.connector.class.state | The name of the connector class. | |
69 | | -| kafka.connect.connector.type | UpDownCounter | 1 | kafka.connect.connector.type | The type of the connector. One of 'source' or 'sink'. | |
70 | | -| kafka.connect.connector.version | UpDownCounter | 1 | kafka.connect.connector.version.state | The version of the connector class, as reported by the connector. | |
71 | | -| kafka.connect.connector.status | UpDownCounter | 1 | kafka.connect.connector.state | Connector lifecycle state indicator (1 when the state matches the attribute value); accepts running, paused, stopped, failed, restarting, unassigned, degraded, or unknown. | |
72 | | - |
73 | | -## Predicate metrics |
74 | | - |
75 | | -Attributes: `kafka.connect.connector`, `kafka.connect.task.id`, `kafka.connect.predicate`, `kafka.connect.predicate.class`, `kafka.connect.predicate.version`. |
| 60 | +Attributes: `kafka.connect.connector` and the state attribute shown. |
76 | 61 |
|
77 | | -| Metric Name | Type | Unit | Attributes | Description | |
78 | | -|---------------------------------|---------------|------|---------------------------------------|----------------------------------------| |
79 | | -| kafka.connect.predicate.class | UpDownCounter | 1 | kafka.connect.predicate.class.state | The class name of the predicate class. | |
80 | | -| kafka.connect.predicate.version | UpDownCounter | 1 | kafka.connect.predicate.version.state | The version of the predicate class. | |
| 62 | +| Metric Name | Type | Unit | Attributes | Description | |
| 63 | +|---------------------------------|---------------|------|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 64 | +| kafka.connect.connector.type | UpDownCounter | 1 | kafka.connect.connector.type | The type of the connector. One of 'source' or 'sink'. | |
| 65 | +| kafka.connect.connector.status | UpDownCounter | 1 | kafka.connect.connector.state | Connector lifecycle state indicator (1 when the state matches the attribute value); accepts running, paused, stopped, failed, restarting, unassigned, degraded, or unknown. | |
81 | 66 |
|
82 | 67 | ## Connector task metrics |
83 | 68 |
|
84 | | -Attributes include `kafka.connect.connector`, `kafka.connect.task.id`, connector class/type/version, converter class/version attributes, and task class/version. |
| 69 | +All metrics include `kafka.connect.connector` and `kafka.connect.task.id`. Attributes column lists any additional state attributes. |
85 | 70 |
|
86 | 71 | | Metric Name | Type | Unit | Attributes | Description | |
87 | 72 | |-----------------------------------------------------|---------------|----------|---------------------------------------------------|--------------------------------------------------------------------------------------------------------------------| |
88 | 73 | | kafka.connect.task.batch.size.avg | Gauge | {record} | | The average number of records in the batches the task has processed so far. | |
89 | 74 | | kafka.connect.task.batch.size.max | Gauge | {record} | | The number of records in the largest batch the task has processed so far. | |
90 | | -| kafka.connect.task.connector.class | UpDownCounter | 1 | kafka.connect.task.connector.class.state | The name of the connector class. | |
91 | 75 | | kafka.connect.task.connector.type | UpDownCounter | 1 | kafka.connect.task.connector.type | The type of the connector. One of 'source' or 'sink'. | |
92 | | -| kafka.connect.task.connector.version | UpDownCounter | 1 | kafka.connect.task.connector.version.state | The version of the connector class, as reported by the connector. | |
93 | | -| kafka.connect.task.header.converter.class | UpDownCounter | 1 | kafka.connect.task.header.converter.class.state | The fully qualified class name from header.converter. | |
94 | | -| kafka.connect.task.header.converter.version | UpDownCounter | 1 | kafka.connect.task.header.converter.version.state | The version instantiated for header.converter. May be undefined. | |
95 | | -| kafka.connect.task.key.converter.class | UpDownCounter | 1 | kafka.connect.task.key.converter.class.state | The fully qualified class name from key.converter. | |
96 | | -| kafka.connect.task.key.converter.version | UpDownCounter | 1 | kafka.connect.task.key.converter.version.state | The version instantiated for key.converter. May be undefined. | |
97 | 76 | | kafka.connect.task.offset.commit.avg.time | Gauge | s | | The average time in milliseconds taken by this task to commit offsets. | |
98 | 77 | | kafka.connect.task.offset.commit.failure.percentage | Gauge | 1 | | The average percentage of this task's offset commit attempts that failed. | |
99 | 78 | | kafka.connect.task.offset.commit.max.time | Gauge | s | | The maximum time in milliseconds taken by this task to commit offsets. | |
100 | 79 | | kafka.connect.task.offset.commit.success.percentage | Gauge | 1 | | The average percentage of this task's offset commit attempts that succeeded. | |
101 | 80 | | kafka.connect.task.pause.ratio | Gauge | 1 | | The fraction of time this task has spent in the pause state. | |
102 | 81 | | kafka.connect.task.running.ratio | Gauge | 1 | | The fraction of time this task has spent in the running state. | |
103 | 82 | | kafka.connect.task.status | UpDownCounter | 1 | kafka.connect.task.state | The status of the connector task; supports running, paused, failed, restarting, unassigned, destroyed, or unknown. | |
104 | | -| kafka.connect.task.class | UpDownCounter | 1 | kafka.connect.task.class.state | The class name of the task. | |
105 | | -| kafka.connect.task.version | UpDownCounter | 1 | kafka.connect.task.version.state | The version of the task. | |
106 | | -| kafka.connect.task.value.converter.class | UpDownCounter | 1 | kafka.connect.task.value.converter.class.state | The fully qualified class name from value.converter. | |
107 | | -| kafka.connect.task.value.converter.version | UpDownCounter | 1 | kafka.connect.task.value.converter.version.state | The version instantiated for value.converter. May be undefined. | |
108 | | - |
109 | | -## Transform metrics |
110 | | - |
111 | | -Attributes: `kafka.connect.connector`, `kafka.connect.task.id`, `kafka.connect.transform`, `kafka.connect.transform.class`, `kafka.connect.transform.version`. |
112 | | - |
113 | | -| Metric Name | Type | Unit | Attributes | Description | |
114 | | -|---------------------------------|---------------|------|---------------------------------------|---------------------------------------------| |
115 | | -| kafka.connect.transform.class | UpDownCounter | 1 | kafka.connect.transform.class.state | The class name of the transformation class. | |
116 | | -| kafka.connect.transform.version | UpDownCounter | 1 | kafka.connect.transform.version.state | The version of the transformation class. | |
117 | 83 |
|
118 | 84 | ## Sink task metrics |
119 | 85 |
|
|
0 commit comments