Skip to content

Commit 6293d68

Browse files
aaaugustine29aaaugustine29
committed
initial commit with changes
1 parent e900aee commit 6293d68

File tree

3 files changed

+940
-0
lines changed

3 files changed

+940
-0
lines changed

‎instrumentation/jmx-metrics/library/kafka-connect.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Kafka Connect Metrics
2+
3+
Here is the list of metrics based on MBeans exposed by Kafka Connect. String-valued JMX
4+
attributes (class/type/version information) are exported as state metrics with value `1` and
5+
carry the raw string value as metric attributes.
6+
7+
## Worker metrics
8+
9+
| Metric Name | Type | Unit | Attributes | Description |
10+
|-------------|------|------|------------|-------------|
11+
| kafka.connect.worker.connector.count | UpDownCounter | {connector} | | The number of connectors run in this worker. |
12+
| kafka.connect.worker.connector.startup.attempts | Counter | {attempt} | | The total number of connector startups that this worker has attempted. |
13+
| kafka.connect.worker.connector.startup.failure.percentage | Gauge | 1 | | The average percentage of this worker's connectors starts that failed. |
14+
| kafka.connect.worker.connector.startup.failure.total | Counter | {startup} | | The total number of connector starts that failed. |
15+
| kafka.connect.worker.connector.startup.success.percentage | Gauge | 1 | | The average percentage of this worker's connectors starts that succeeded. |
16+
| kafka.connect.worker.connector.startup.success.total | Counter | {startup} | | The total number of connector starts that succeeded. |
17+
| kafka.connect.worker.task.count | UpDownCounter | {task} | | The number of tasks run in this worker. |
18+
| kafka.connect.worker.task.startup.attempts | Counter | {attempt} | | The total number of task startups that this worker has attempted. |
19+
| kafka.connect.worker.task.startup.failure.percentage | Gauge | 1 | | The average percentage of this worker's tasks starts that failed. |
20+
| kafka.connect.worker.task.startup.failure.total | Counter | {startup} | | The total number of task starts that failed. |
21+
| kafka.connect.worker.task.startup.success.percentage | Gauge | 1 | | The average percentage of this worker's tasks starts that succeeded. |
22+
| kafka.connect.worker.task.startup.success.total | Counter | {startup} | | The total number of task starts that succeeded. |
23+
24+
## Worker connector task metrics
25+
26+
| Metric Name | Type | Unit | Attributes | Description |
27+
|-------------|------|------|------------|-------------|
28+
| kafka.connect.worker.connector.task.destroyed | UpDownCounter | {task} | kafka.connect.connector | The number of destroyed tasks of the connector on the worker. |
29+
| kafka.connect.worker.connector.task.failed | UpDownCounter | {task} | kafka.connect.connector | The number of failed tasks of the connector on the worker. |
30+
| kafka.connect.worker.connector.task.paused | UpDownCounter | {task} | kafka.connect.connector | The number of paused tasks of the connector on the worker. |
31+
| kafka.connect.worker.connector.task.restarting | UpDownCounter | {task} | kafka.connect.connector | The number of restarting tasks of the connector on the worker. |
32+
| kafka.connect.worker.connector.task.running | UpDownCounter | {task} | kafka.connect.connector | The number of running tasks of the connector on the worker. |
33+
| kafka.connect.worker.connector.task.total | UpDownCounter | {task} | kafka.connect.connector | The number of tasks of the connector on the worker. |
34+
| kafka.connect.worker.connector.task.unassigned | UpDownCounter | {task} | kafka.connect.connector | The number of unassigned tasks of the connector on the worker. |
35+
36+
## Worker rebalance metrics
37+
38+
All metrics include `kafka.connect.worker.leader`.
39+
40+
| Metric Name | Type | Unit | Attributes | Description |
41+
|-------------|------|------|------------|-------------|
42+
| kafka.connect.worker.rebalance.completed.total | Counter | {rebalance} | | The total number of rebalances completed by this worker. |
43+
| kafka.connect.worker.rebalance.protocol | UpDownCounter | 1 | kafka.connect.protocol.state | The Connect protocol used by this cluster. |
44+
| kafka.connect.worker.rebalance.epoch | UpDownCounter | {epoch} | | The epoch or generation number of this worker. |
45+
| kafka.connect.worker.rebalance.leader | UpDownCounter | 1 | kafka.connect.worker.leader.state | The name of the group leader. |
46+
| kafka.connect.worker.rebalance.avg.time | Gauge | s | | The average time in milliseconds spent by this worker to rebalance. |
47+
| kafka.connect.worker.rebalance.max.time | Gauge | s | | The maximum time in milliseconds spent by this worker to rebalance. |
48+
| kafka.connect.worker.rebalance.active | UpDownCounter | 1 | kafka.connect.worker.rebalance.state | Whether this worker is currently rebalancing. |
49+
| kafka.connect.worker.rebalance.since_last | Gauge | s | | The time in milliseconds since this worker completed the most recent rebalance. |
50+
51+
## Connector metrics
52+
53+
Attributes: `kafka.connect.connector`, `kafka.connect.connector.class`, `kafka.connect.connector.version`, `kafka.connect.connector.type.raw`.
54+
55+
| Metric Name | Type | Unit | Attributes | Description |
56+
|-------------|------|------|------------|-------------|
57+
| kafka.connect.connector.class | UpDownCounter | 1 | kafka.connect.connector.class.state | The name of the connector class. |
58+
| kafka.connect.connector.type | UpDownCounter | 1 | kafka.connect.connector.type | The type of the connector. One of 'source' or 'sink'. |
59+
| kafka.connect.connector.version | UpDownCounter | 1 | kafka.connect.connector.version.state | The version of the connector class, as reported by the connector. |
60+
| kafka.connect.connector.status | UpDownCounter | 1 | kafka.connect.connector.state | Connector lifecycle state indicator (1 when the state matches the attribute value). |
61+
62+
## Predicate metrics
63+
64+
Attributes: `kafka.connect.connector`, `kafka.connect.task.id`, `kafka.connect.predicate`, `kafka.connect.predicate.class`, `kafka.connect.predicate.version`.
65+
66+
| Metric Name | Type | Unit | Attributes | Description |
67+
|-------------|------|------|------------|-------------|
68+
| kafka.connect.predicate.class | UpDownCounter | 1 | kafka.connect.predicate.class.state | The class name of the predicate class. |
69+
| kafka.connect.predicate.version | UpDownCounter | 1 | kafka.connect.predicate.version.state | The version of the predicate class. |
70+
71+
## Connector task metrics
72+
73+
Attributes include `kafka.connect.connector`, `kafka.connect.task.id`, connector class/type/version, converter class/version attributes, and task class/version.
74+
75+
| Metric Name | Type | Unit | Attributes | Description |
76+
|-------------|------|------|------------|-------------|
77+
| kafka.connect.task.batch.size.avg | Gauge | {record} | | The average number of records in the batches the task has processed so far. |
78+
| kafka.connect.task.batch.size.max | Gauge | {record} | | The number of records in the largest batch the task has processed so far. |
79+
| kafka.connect.task.connector.class | UpDownCounter | 1 | kafka.connect.task.connector.class.state | The name of the connector class. |
80+
| kafka.connect.task.connector.type | UpDownCounter | 1 | kafka.connect.task.connector.type | The type of the connector. One of 'source' or 'sink'. |
81+
| kafka.connect.task.connector.version | UpDownCounter | 1 | kafka.connect.task.connector.version.state | The version of the connector class, as reported by the connector. |
82+
| kafka.connect.task.header.converter.class | UpDownCounter | 1 | kafka.connect.task.header.converter.class.state | The fully qualified class name from header.converter. |
83+
| kafka.connect.task.header.converter.version | UpDownCounter | 1 | kafka.connect.task.header.converter.version.state | The version instantiated for header.converter. May be undefined. |
84+
| kafka.connect.task.key.converter.class | UpDownCounter | 1 | kafka.connect.task.key.converter.class.state | The fully qualified class name from key.converter. |
85+
| kafka.connect.task.key.converter.version | UpDownCounter | 1 | kafka.connect.task.key.converter.version.state | The version instantiated for key.converter. May be undefined. |
86+
| kafka.connect.task.offset.commit.avg.time | Gauge | s | | The average time in milliseconds taken by this task to commit offsets. |
87+
| kafka.connect.task.offset.commit.failure.percentage | Gauge | 1 | | The average percentage of this task's offset commit attempts that failed. |
88+
| kafka.connect.task.offset.commit.max.time | Gauge | s | | The maximum time in milliseconds taken by this task to commit offsets. |
89+
| kafka.connect.task.offset.commit.success.percentage | Gauge | 1 | | The average percentage of this task's offset commit attempts that succeeded. |
90+
| kafka.connect.task.pause.ratio | Gauge | 1 | | The fraction of time this task has spent in the pause state. |
91+
| kafka.connect.task.running.ratio | Gauge | 1 | | The fraction of time this task has spent in the running state. |
92+
| kafka.connect.task.status | UpDownCounter | 1 | kafka.connect.task.state | The status of the connector task. |
93+
| kafka.connect.task.class | UpDownCounter | 1 | kafka.connect.task.class.state | The class name of the task. |
94+
| kafka.connect.task.version | UpDownCounter | 1 | kafka.connect.task.version.state | The version of the task. |
95+
| kafka.connect.task.value.converter.class | UpDownCounter | 1 | kafka.connect.task.value.converter.class.state | The fully qualified class name from value.converter. |
96+
| kafka.connect.task.value.converter.version | UpDownCounter | 1 | kafka.connect.task.value.converter.version.state | The version instantiated for value.converter. May be undefined. |
97+
98+
## Transform metrics
99+
100+
Attributes: `kafka.connect.connector`, `kafka.connect.task.id`, `kafka.connect.transform`, `kafka.connect.transform.class`, `kafka.connect.transform.version`.
101+
102+
| Metric Name | Type | Unit | Attributes | Description |
103+
|-------------|------|------|------------|-------------|
104+
| kafka.connect.transform.class | UpDownCounter | 1 | kafka.connect.transform.class.state | The class name of the transformation class. |
105+
| kafka.connect.transform.version | UpDownCounter | 1 | kafka.connect.transform.version.state | The version of the transformation class. |
106+
107+
## Sink task metrics
108+
109+
Attributes: `kafka.connect.connector`, `kafka.connect.task.id`.
110+
111+
| Metric Name | Type | Unit | Attributes | Description |
112+
|-------------|------|------|------------|-------------|
113+
| kafka.connect.sink.offset.commit.completion.rate | Gauge | {commit}/s | | The average per-second number of offset commit completions that were completed successfully. |
114+
| kafka.connect.sink.offset.commit.completion.total | Counter | {commit} | | The total number of offset commit completions that were completed successfully. |
115+
| kafka.connect.sink.offset.commit.seq | UpDownCounter | {sequence} | | The current sequence number for offset commits. |
116+
| kafka.connect.sink.offset.commit.skip.rate | Gauge | {commit}/s | | The average per-second number of offset commit completions that were received too late and skipped/ignored. |
117+
| kafka.connect.sink.offset.commit.skip.total | Counter | {commit} | | The total number of offset commit completions that were received too late and skipped/ignored. |
118+
| kafka.connect.sink.partition.count | UpDownCounter | {partition} | | The number of topic partitions assigned to this task. |
119+
| kafka.connect.sink.put.batch.avg.time | Gauge | s | | The average time taken by this task to put a batch of sinks records. |
120+
| kafka.connect.sink.put.batch.max.time | Gauge | s | | The maximum time taken by this task to put a batch of sinks records. |
121+
| kafka.connect.sink.record.active.count | UpDownCounter | {record} | | The number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task. |
122+
| kafka.connect.sink.record.active.count.avg | Gauge | {record} | | The average number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task. |
123+
| kafka.connect.sink.record.active.count.max | Gauge | {record} | | The maximum number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task. |
124+
| kafka.connect.sink.record.lag.max | Gauge | {record} | | The maximum lag in terms of number of records that the sink task is behind the consumer's position for any topic partitions. |
125+
| kafka.connect.sink.record.read.rate | Gauge | {record}/s | | The average per-second number of records read from Kafka for this task before transformations are applied. |
126+
| kafka.connect.sink.record.read.total | Counter | {record} | | The total number of records read from Kafka by this task since it was last restarted. |
127+
| kafka.connect.sink.record.send.rate | Gauge | {record}/s | | The average per-second number of records output from the transformations and sent/put to this task. |
128+
| kafka.connect.sink.record.send.total | Counter | {record} | | The total number of records output from the transformations and sent/put to this task since it was last restarted. |
129+
130+
## Source task metrics
131+
132+
Attributes: `kafka.connect.connector`, `kafka.connect.task.id`.
133+
134+
| Metric Name | Type | Unit | Attributes | Description |
135+
|-------------|------|------|------------|-------------|
136+
| kafka.connect.source.poll.batch.avg.time | Gauge | s | | The average time in milliseconds taken by this task to poll for a batch of source records. |
137+
| kafka.connect.source.poll.batch.max.time | Gauge | s | | The maximum time in milliseconds taken by this task to poll for a batch of source records. |
138+
| kafka.connect.source.record.active.count | UpDownCounter | {record} | | The number of records that have been produced by this task but not yet completely written to Kafka. |
139+
| kafka.connect.source.record.active.count.avg | Gauge | {record} | | The average number of records that have been produced by this task but not yet completely written to Kafka. |
140+
| kafka.connect.source.record.active.count.max | Gauge | {record} | | The maximum number of records that have been produced by this task but not yet completely written to Kafka. |
141+
| kafka.connect.source.record.poll.rate | Gauge | {record}/s | | The average per-second number of records produced/polled (before transformation) by this task. |
142+
| kafka.connect.source.record.poll.total | Counter | {record} | | The total number of records produced/polled (before transformation) by this task. |
143+
| kafka.connect.source.record.write.rate | Gauge | {record}/s | | The average per-second number of records written to Kafka for this task. |
144+
| kafka.connect.source.record.write.total | Counter | {record} | | The number of records output written to Kafka for this task. |
145+
| kafka.connect.source.transaction.size.avg | Gauge | {record} | | The average number of records in the transactions the task has committed so far. |
146+
| kafka.connect.source.transaction.size.max | Gauge | {record} | | The number of records in the largest transaction the task has committed so far. |
147+
| kafka.connect.source.transaction.size.min | Gauge | {record} | | The number of records in the smallest transaction the task has committed so far. |
148+
149+
## Task error metrics
150+
151+
Attributes: `kafka.connect.connector`, `kafka.connect.task.id`.
152+
153+
| Metric Name | Type | Unit | Attributes | Description |
154+
|-------------|------|------|------------|-------------|
155+
| kafka.connect.task.error.deadletterqueue.produce.failures | Counter | {failure} | | The number of failed writes to the dead letter queue. |
156+
| kafka.connect.task.error.deadletterqueue.produce.requests | Counter | {request} | | The number of attempted writes to the dead letter queue. |
157+
| kafka.connect.task.error.last.error.timestamp | Gauge | s | | The epoch timestamp when this task last encountered an error. |
158+
| kafka.connect.task.error.total.errors.logged | Counter | {error} | | The number of errors that were logged. |
159+
| kafka.connect.task.error.total.record.errors | Counter | {record} | | The number of record processing errors in this task. |
160+
| kafka.connect.task.error.total.record.failures | Counter | {record} | | The number of record processing failures in this task. |
161+
| kafka.connect.task.error.total.records.skipped | Counter | {record} | | The number of records skipped due to errors. |
162+
| kafka.connect.task.error.total.retries | Counter | {retry} | | The number of operations retried. |

0 commit comments

Comments
 (0)