Skip to content

Commit 87dd373

Browse files
gnuhpcgnuhpc
authored
Put an end to the 0.10.x (#25)
Put an end to the 0.10.x
2 parents d9fec3e + f6e9aa5 commit 87dd373

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+12385
-6847
lines changed

‎.travis.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
language: java
2+
jdk: oraclejdk8
3+
install:
4+
- mvn -N io.takari:maven:wrapper
5+
- ./mvnw install -DskipTests=true -Dmaven.javadoc.skip=true -B -V
6+
script:
7+
- echo "skipping tests"
8+
before_install:
9+
- chmod +x mvnw

‎JMXFilterTemplate/KafkaBrokerFilter.yml‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
filters:
2+
#
3+
# Kafka volume metrics.
4+
# Notice: BytesInPerSec, BytesOutPerSec and MessagesInPerSec are performance metrics.
5+
#
6+
- include:
7+
domain: kafka.server
8+
bean: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
9+
attribute:
10+
Count:
11+
metric_type: KAFKA_BROKER_VOLUMN
12+
alias: BytesIn
13+
OneMinuteRate:
14+
metric_type: KAFKA_BROKER_PERF
15+
alias: BytesInPerSec
16+
- include:
17+
domain: kafka.server
18+
bean: kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
19+
attribute:
20+
Count:
21+
metric_type: KAFKA_BROKER_VOLUMN
22+
alias: BytesOut
23+
OneMinuteRate:
24+
metric_type: KAFKA_BROKER_PERF
25+
alias: BytesOutPerSec
26+
- include:
27+
domain: kafka.server
28+
bean: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
29+
attribute:
30+
Count:
31+
metric_type: KAFKA_BROKER_VOLUMN
32+
alias: MessagesIn
33+
OneMinuteRate:
34+
metric_type: KAFKA_BROKER_PERF
35+
alias: MessagesInPerSec
36+
37+
#
38+
# Kafka performance metrics
39+
#
40+
- include:
41+
domain: kafka.network
42+
bean: kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce
43+
attribute:
44+
OneMinuteRate:
45+
metric_type: KAFKA_BROKER_PERF
46+
alias: RequestsPerSec_Produce
47+
- include:
48+
domain: kafka.network
49+
bean: kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer
50+
attribute:
51+
OneMinuteRate:
52+
metric_type: KAFKA_BROKER_PERF
53+
alias: RequestsPerSec_FetchConsumer
54+
- include:
55+
domain: kafka.network
56+
bean: kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower
57+
attribute:
58+
OneMinuteRate:
59+
metric_type: KAFKA_BROKER_PERF
60+
alias: RequestsPerSec_FetchFollower
61+
- include:
62+
domain: kafka.network
63+
bean: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
64+
attribute:
65+
Mean:
66+
metric_type: KAFKA_BROKER_PERF
67+
alias: TotalTimeMs_Produce_Mean
68+
Count:
69+
metric_type: KAFKA_BROKER_PERF
70+
alias: TotalTimeMs_Produce_Count
71+
- include:
72+
domain: kafka.network
73+
bean: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer
74+
attribute:
75+
Mean:
76+
metric_type: KAFKA_BROKER_PERF
77+
alias: TotalTimeMs_FetchConsumer_Mean
78+
Count:
79+
metric_type: KAFKA_BROKER_PERF
80+
alias: TotalTimeMs_FetchConsumer_Count
81+
- include:
82+
domain: kafka.network
83+
bean: kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower
84+
attribute:
85+
Mean:
86+
metric_type: KAFKA_BROKER_PERF
87+
alias: TotalTimeMs_FetchFollower_Mean
88+
Count:
89+
metric_type: KAFKA_BROKER_PERF
90+
alias: TotalTimeMs_FetchFollower_Count
91+
- include:
92+
domain: kafka.network
93+
bean: kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent
94+
attribute:
95+
Value:
96+
metric_type: KAFKA_BROKER_PERF
97+
alias: NetworkProcessorAvgIdlePercent
98+
- include:
99+
domain: kafka.server
100+
bean: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent
101+
attribute:
102+
FiveMinuteRate:
103+
metric_type: KAFKA_BROKER_PERF
104+
alias: RequestHandlerAvgIdlePercent
105+
- include:
106+
domain: kafka.server
107+
bean: kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica
108+
attribute:
109+
Value:
110+
metric_type: KAFKA_BROKER_PERF
111+
alias: MaxLagBtwFollowerLeader
112+
#
113+
# Kafka availability metrics
114+
#
115+
- include:
116+
domain: java.lang
117+
bean: java.lang:type=Runtime
118+
attribute:
119+
Uptime:
120+
metric_type: KAFKA_BROKER_AVAIL
121+
alias: JVMUptime
122+
- include:
123+
domain: kafka.controller
124+
bean: kafka.controller:type=KafkaController,name=OfflinePartitionsCount
125+
attribute:
126+
Value:
127+
metric_type: KAFKA_BROKER_AVAIL
128+
alias: OfflinePartitionsCount
129+
- include:
130+
domain: kafka.controller
131+
bean: kafka.controller:type=KafkaController,name=ActiveControllerCount
132+
attribute:
133+
Value:
134+
metric_type: KAFKA_BROKER_AVAIL
135+
alias: ActiveControllerCount
136+
- include:
137+
domain: kafka.controller
138+
bean: kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
139+
attribute:
140+
Count:
141+
metric_type: KAFKA_BROKER_AVAIL
142+
alias: LeaderElectionRateAndTimeMs
143+
- include:
144+
domain: kafka.controller
145+
bean: kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec
146+
attribute:
147+
Count:
148+
metric_type: KAFKA_BROKER_AVAIL
149+
alias: UncleanLeaderElectionsPerSec
150+
- include:
151+
domain: kafka.server
152+
bean: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
153+
attribute:
154+
Value:
155+
metric_type: KAFKA_BROKER_AVAIL
156+
alias: UnderReplicatedPartitions
157+
- include:
158+
domain: kafka.server
159+
bean: kafka.server:type=ReplicaManager,name=PartitionCount
160+
attribute:
161+
Value:
162+
metric_type: KAFKA_BROKER_AVAIL
163+
alias: PartitionCount
164+
- include:
165+
domain: kafka.server
166+
bean: kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
167+
attribute:
168+
Count:
169+
metric_type: KAFKA_BROKER_AVAIL
170+
alias: IsrShrinksPerSec
171+
- include:
172+
domain: kafka.server
173+
bean: kafka.server:type=ReplicaManager,name=IsrExpandsPerSec
174+
attribute:
175+
Count:
176+
metric_type: KAFKA_BROKER_AVAIL
177+
alias: IsrExpandsPerSec
178+
- include:
179+
domain: kafka.server
180+
bean: kafka.server:type=ReplicaManager,name=LeaderCount
181+
attribute:
182+
Value:
183+
metric_type: KAFKA_BROKER_AVAIL
184+
alias: LeaderCount

‎JMXFilterTemplate/KafkaConsumerFilter.yml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
filters:
2+
- include:
3+
domain: kafka.consumer
4+
bean_regex: kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)
5+
attribute:
6+
Value:
7+
metric_type: KAFKA_CONSUMER_OLD_HIGH
8+
alias: MaxLag
9+
- include:
10+
domain: kafka.consumer
11+
bean_regex: kafka.consumer:type=ConsumerFetcherManager,name=MinFetchRate,clientId=([-.\w]+)
12+
attribute:
13+
Value:
14+
metric_type: KAFKA_CONSUMER_OLD_HIGH
15+
alias: MinFetchRate
16+
- include:
17+
domain: kafka.consumer
18+
bean_regex: kafka.consumer:type=ConsumerTopicMetrics,name=MessagesPerSec,clientId=([-.\w]+)
19+
attribute:
20+
Count:
21+
metric_type: KAFKA_CONSUMER_OLD_HIGH
22+
alias: MessagesPerSec
23+
- include:
24+
domain: kafka.consumer
25+
bean_regex: kafka.consumer:type=ConsumerTopicMetrics,name=BytesPerSec,clientId=([-.\w]+)
26+
attribute:
27+
Count:
28+
metric_type: KAFKA_CONSUMER_OLD_HIGH
29+
alias: BytesPerSec
30+
- include:
31+
domain: kafka.consumer
32+
bean_regex: kafka.consumer:type=ZookeeperConsumerConnector,name=KafkaCommitsPerSec,clientId=([-.\w]+)
33+
attribute:
34+
Count:
35+
metric_type: KAFKA_CONSUMER_OLD_HIGH
36+
alias: KafkaCommitsPerSec
37+
- include:
38+
domain: kafka.consumer
39+
bean_regex: kafka.consumer:type=ZookeeperConsumerConnector,name=OwnedPartitionsCount,clientId=([-.\w]+),groupId=([-.\w]+)
40+
attribute:
41+
Value:
42+
metric_type: KAFKA_CONSUMER_OLD_HIGH
43+
alias: OwnedPartitionsCount

‎JMXFilterTemplate/KafkaProducerFilter.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
filters:
2+
- include:
3+
domain: kafka.producer
4+
bean_regex: kafka.producer:type=producer-metrics,client-id=([-.\w]+)
5+
attribute:
6+
request-rate:
7+
metric_type: KAFKA_PRODUCER
8+
alias: request-rate
9+
request-size-avg:
10+
metric_type: KAFKA_PRODUCER
11+
alias: request-size-avg
12+
# There is some problem. I can't find producer-topic-metrics in Kafka 0.10
13+
- include:
14+
domain: kafka.producer
15+
bean_regex: kafka.producer:type=producer-topic-metrics,client-id=([-.\w]+)
16+
attribute:
17+
byte-rate:
18+
metric_type: KAFKA_PRODUCER
19+
alias: byte-rate-topic
20+

‎JMXFilterTemplate/KafkaTopicFilter.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
filters:
2+
- include:
3+
domain: kafka.server
4+
bean: kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic={topicname}
5+
attribute:
6+
Count:
7+
metric_type: KAFKA_TOPIC_PERF
8+
alias: BytesInPerSec
9+
- include:
10+
domain: kafka.server
11+
bean: kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic={topicname}
12+
attribute:
13+
Count:
14+
metric_type: KAFKA_TOPIC_PERF
15+
alias: BytesOutPerSec
16+
- include:
17+
domain: kafka.server
18+
bean: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic={topicname}
19+
attribute:
20+
Count:
21+
metric_type: KAFKA_TOPIC_PERF
22+
alias: MessagesInPerSec
23+
- include:
24+
domain: kafka.server
25+
bean: kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic={topicname}
26+
attribute:
27+
Count:
28+
metric_type: KAFKA_TOPIC_PERF
29+
alias: BytesRejectedPerSec
30+
- include:
31+
domain: kafka.server
32+
bean: kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec,topic={topicname}
33+
attribute:
34+
Count:
35+
metric_type: KAFKA_TOPIC_PERF
36+
alias: FailedFetchRequestsPerSec
37+
- include:
38+
domain: kafka.server
39+
bean: kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec,topic={topicname}
40+
attribute:
41+
Count:
42+
metric_type: KAFKA_TOPIC_PERF
43+
alias: FailedProduceRequestsPerSec
44+
- include:
45+
domain: kafka.server
46+
bean: kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic={topicname}
47+
attribute:
48+
Count:
49+
metric_type: KAFKA_TOPIC_PERF
50+
alias: TotalFetchRequestsPerSec
51+
- include:
52+
domain: kafka.server
53+
bean: kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic={topicname}
54+
attribute:
55+
Count:
56+
metric_type: KAFKA_TOPIC_PERF
57+
alias: TotalProduceRequestsPerSec

‎README.adoc‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
= Kafka REST API
22

3+
image:https://travis-ci.org/gnuhpc/Kafka-zk-restapi.svg?branch=master["Build Status", link="https://travis-ci.org/gnuhpc/Kafka-zk-restapi"]
4+
35
[[_overview]]
46
== Overview
57
Kafka/ZK REST API is to provide the production-ready endpoints to perform some administration/metric task for Kafka and Zookeeper.
@@ -13,6 +15,9 @@ Kafka/ZK REST API is to provide the production-ready endpoints to perform some a
1315
* Consumer group(old zookeeper based/new kafka based) list/describe
1416
* Offset check/reset
1517
* Consumer Group Lag check
18+
* Collect JMX metrics from brokers that expose JMX metrics +
19+
More details refer to https://github.com/gnuhpc/Kafka-zk-restapi/blob/master/docs/JMXCollector.adoc[JMXCollector API Specification]
20+
* Secure the REST API with Spring Security
1621
// end::base-t[]
1722

1823
image::https://raw.githubusercontent.com/gnuhpc/Kafka-zk-restapi/master/pics/ShowApi.png[API]
@@ -36,14 +41,48 @@ Change the following settings of application-home.yml in src/main/resources to v
3641
kafka.brokers
3742
zookeeper.uris
3843

44+
Change security related parameters in application config file:
45+
See below: <<_security,'How to config security'>>
46+
47+
server.security.check
48+
server.security.checkInitDelay
49+
server.security.checkSecurityInterval
50+
51+
If you want to use JMX Query Filter function, you can add your own custom filter files to JMXFilterTemplate directory in project root folder.
52+
More details refer to https://github.com/gnuhpc/Kafka-zk-restapi/blob/master/docs/JMXCollector.adoc[JMXCollector API Specification]
53+
3954
#mvn clean package -Dmaven.test.skip=true+
4055

4156
You will find zip/tar under directory: Kafka-zk-restapi/target
4257

4358
You can get it running by unzip/untaring the distribution package and run bin/start.sh
4459

60+
[[_security]]
61+
=== Security
62+
Public REST services without access control make the sensitive data under risk.Then we provide a simple authentication mechanism using Spring Security.
63+
In order to make the project lighter, we use yml file to store user information, not using database.
64+
65+
Follow the steps to enable security feature：
66+
67+
Step 1：Modify the application config file and set server.security.check to true. +
68+
69+
* server.security.check:
70+
** True: Add security for the API. Clients can access the API with valid username and password stored in security.yml, or the Swagger UI(http://127.0.0.1:8121/api) is only
71+
allowed to access.
72+
** False: All the endpoints can be accessed without authentication.
73+
* server.security.checkInitDelay: The number of seconds of init delay for the timing thread to check the security file.
74+
* server.security.checkSecurityInterval: The number of seconds of check interval for the timing thread to check the security file.
75+
76+
Step 2: Make sure security/security.yml exist in application root folder.
77+
78+
Step 3: Use user controller API to add user to security file security/security.yml. +
79+
**Notice**:
80+
81+
* The first user should be added manually. Password need to be encoded using bcrypt before saving to the yml file.For convenience, we provide CommonUtils to encode the password.
82+
* No need to restart server after adding new user or update user info. Timing thread introduced in Step 1 will refresh the user list according to your settings.
83+
4584
=== Support Kafka Version Information
46-
Currently, this rest api (master branch) supports Kafka 0.10.x brokers. The master branch is the most active branch.
85+
Currently, this rest api (master branch) supports Kafka 0.10.x brokers. The master branch is the most active branch. We're going to get down to the work of supporting the Kafka 1.x version.
4786

4887
*For 0.11.x, please checkout the branch 0.11.x by calling the command:*
4988

@@ -63,6 +102,8 @@ You can access Swagger-UI by accessing http://127.0.0.1:8121/api
63102

64103
* kafka-controller : Kafka Api
65104
* zookeeper-controller : Zookeeper Api
105+
* collector-controller : JMX Metric Collector Api
106+
* user-controller : User management Api
66107

67108

68109
=== https://github.com/gnuhpc/Kafka-zk-restapi/blob/master/docs/definitions.adoc[Data Model Definitions for 0.10]

‎_config.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
theme: jekyll-theme-cayman

0 commit comments

Comments
 (0)