Skip to content

Commit e72f6d6

Browse files
David-DB88GneyHabubworkshuriliaxiliax
authored
Feature: Cluster web configuration wizard (#3241)
* created wizard * Create wizard form schema * Wizard kafka cluster form (#3245) * created wizard Kafka Cluster form * created error message Co-authored-by: davitbejanyan <[email protected]> * Update schema.ts * Wizard authentication (#3268) * created authentication form * changed SaslType.tsx switch case * remove console.log * commented unused variables * auth validation * auth Security Protocol * changed schema.ts username, password * added Delegation tokens validation schema * changed auth form --------- Co-authored-by: davitbejanyan <[email protected]> * New Dashboard flow. Add an ability to configure clusters * wizard kafka cluster validate (#3294) * kafka cluster validate * fixed bootstrap server uncontrolled input warning error --------- Co-authored-by: davitbejanyan <[email protected]> * Wizard schema registry (#3286) * created schema registry * unused variables * Prevent Default on click --------- Co-authored-by: davitbejanyan <[email protected]> * feat: cleanup * Application config API (#3242) * wip * wip * wip * wip * OAuthProperties added to dynamic config api * wip * files upload endpoint added * rbac conf api added * rbac conf api improvements * App configuration validation endpoint (#3264) Co-authored-by: iliax <[email protected]> --------- Co-authored-by: iliax <[email protected]> Co-authored-by: Oleg Shur <[email protected]> * add app config api client * refactor cluster section * refactor cluster section * linting * refactor Registry Form (#3311) * refactor Registry Form * refactor Registry --------- Co-authored-by: davitbejanyan <[email protected]> * auth form improvements * refactoring * linting * file upload API changes * Auth * Start connecting to schema & payload * Auth * fileupload * Wizard JMX Metrics form (#3303) * created JMX Metrics form * refactor JMXMetrics.tsx styles * added cursor on checkbox, changed styles submit button * refactor Metrics * refactoring * uncomment schema connect validation --------- Co-authored-by: davitbejanyan <[email protected]> * validate api * refactor * Wizard Kafka Connect form (#3300) * created Kafka Connect form * renaming functions and variables * refactor * changed button name * refactoring kafka connect * made handler function, reset replace with set value, * refactoring * uncomment schema metrics validation --------- Co-authored-by: davitbejanyan <[email protected]> * fixing AdminClient validation * fixing AdminClient validation * refactor kafka connect * refactor metrics * Per-cluster SSL verification settings (#3336) * ssl configuration moved to app & cluster level * documentations changes * trust all removed, global app settings removed * extracting ssl properties settings to SslPropertiesUtil * wip * documentation fix --------- Co-authored-by: iliax <[email protected]> Co-authored-by: Roman Zabaluev <[email protected]> * SSL properties NPE fixes * api integration * custom fields for existing auth config * OffsetsResetServiceTest fix * cluster.properties structure flattening added * kafka-ssl.yml: ssl properties moved to separate section, producer ssl properties copy added * custom auth * error messaging * form submit * feedback * 1. defaulting metrics type to JMX 2. AdminClient id generation made uniq * checkstyle fix * checkstyle fix * refactoring * feedback * feedback * feedback * feedback * feedback * feedback * Wizard: Application info API (#3391) * Application info API added, cluster features enum renamed to `ClusterFeature` * show config for specific envs only * refactor widget * Cluster connection validation err msg improved * KSQL DB section * Refactor + deps upgrade * experiment: get rid of babel * BE validations refactoring * Update kafka-ui.yaml fixed to string type param * fixes #3397 * linting * #3399 adjust size of port input * disable selects for disabled form * Wizard: Keystore separation (#3425) * wip * wip * compose fix * dto structure fix --------- Co-authored-by: iliax <[email protected]> * dynamic ops enablement properties improvements * own keystore for each section * linting * fix keystore submit * fix keystore submit * feedback * feedback * refactoring * Connect config userName field renamed * metrics configs mapping fix * feedback * Wizard: Jmx ssl (#3448) JMX SSL implementation. Added ability to set specific ssl keystore for each cluster when connection to jmx endpoint. * Review fixes * upd compareVersionsOperation qase id * add toBeAutomated into manual suite * DYNAMIC_CONFIG_ENABLED property description added * Resolve conflicts * Fix issue with 400 error * fix SR edit form --------- Co-authored-by: davitbejanyan <[email protected]> Co-authored-by: Alexander Krivonosov <[email protected]> Co-authored-by: Oleg Shur <[email protected]> Co-authored-by: Ilya Kuramshin <[email protected]> Co-authored-by: iliax <[email protected]> Co-authored-by: Roman Zabaluev <[email protected]> Co-authored-by: bkhakimov <[email protected]> Co-authored-by: Mgrdich <[email protected]> Co-authored-by: VladSenyuta <[email protected]>
1 parent 334ba3d commit e72f6d6

File tree

165 files changed

+5324
-4148
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

165 files changed

+5324
-4148
lines changed

‎README.md‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -185,16 +185,17 @@ For example, if you want to use an environment variable to set the `name` parame
185185
|`KAFKA_CLUSTERS_0_KSQLDBSERVERAUTH_PASSWORD` | KSQL DB server's basic authentication password
186186
|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTORELOCATION` |Path to the JKS keystore to communicate to KSQL DB
187187
|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTOREPASSWORD` |Password of the JKS keystore for KSQL DB
188-
|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTORELOCATION` |Path to the JKS truststore to communicate to KSQL DB
189-
|`KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTOREPASSWORD` |Password of the JKS truststore for KSQL DB
190188
|`KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL` |Security protocol to connect to the brokers. For SSL connection use "SSL", for plaintext connection don't set this environment variable
191189
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRY` |SchemaRegistry's address
192190
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYAUTH_USERNAME` |SchemaRegistry's basic authentication username
193191
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYAUTH_PASSWORD` |SchemaRegistry's basic authentication password
194192
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTORELOCATION` |Path to the JKS keystore to communicate to SchemaRegistry
195193
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTOREPASSWORD` |Password of the JKS keystore for SchemaRegistry
196-
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTORELOCATION` |Path to the JKS truststore to communicate to SchemaRegistry
197-
|`KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTOREPASSWORD` |Password of the JKS truststore for SchemaRegistry
194+
|`KAFKA_CLUSTERS_0_METRICS_SSL` |Enable SSL for Metrics (for PROMETHEUS metrics type). Default: false.
195+
|`KAFKA_CLUSTERS_0_METRICS_USERNAME` |Username for Metrics authentication
196+
|`KAFKA_CLUSTERS_0_METRICS_PASSWORD` |Password for Metrics authentication
197+
|`KAFKA_CLUSTERS_0_METRICS_KEYSTORELOCATION` |Path to the JKS keystore to communicate to metrics source (JMX/PROMETHEUS). For advanced setup, see `kafka-ui-jmx-secured.yml`
198+
|`KAFKA_CLUSTERS_0_METRICS_KEYSTOREPASSWORD` |Password of the JKS metrics keystore
198199
|`KAFKA_CLUSTERS_0_SCHEMANAMETEMPLATE` |How keys are saved to schemaRegistry
199200
|`KAFKA_CLUSTERS_0_METRICS_PORT` |Open metrics port of a broker
200201
|`KAFKA_CLUSTERS_0_METRICS_TYPE` |Type of metrics retriever to use. Valid values are JMX (default) or PROMETHEUS. If Prometheus, then metrics are read from prometheus-jmx-exporter instead of jmx
@@ -205,11 +206,9 @@ For example, if you want to use an environment variable to set the `name` parame
205206
|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_PASSWORD`| Kafka Connect cluster's basic authentication password
206207
|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTORELOCATION`| Path to the JKS keystore to communicate to Kafka Connect
207208
|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTOREPASSWORD`| Password of the JKS keystore for Kafka Connect
208-
|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTORELOCATION`| Path to the JKS truststore to communicate to Kafka Connect
209-
|`KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTOREPASSWORD`| Password of the JKS truststore for Kafka Connect
210-
|`KAFKA_CLUSTERS_0_METRICS_SSL` |Enable SSL for Metrics? `true` or `false`. For advanced setup, see `kafka-ui-jmx-secured.yml`
211-
|`KAFKA_CLUSTERS_0_METRICS_USERNAME` |Username for Metrics authentication
212-
|`KAFKA_CLUSTERS_0_METRICS_PASSWORD` |Password for Metrics authentication
213209
|`KAFKA_CLUSTERS_0_POLLING_THROTTLE_RATE` |Max traffic rate (bytes/sec) that kafka-ui allowed to reach when polling messages from the cluster. Default: 0 (not limited)
210+
|`KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION`| Path to the JKS truststore to communicate to Kafka Connect, SchemaRegistry, KSQL, Metrics
211+
|`KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD`| Password of the JKS truststore for Kafka Connect, SchemaRegistry, KSQL, Metrics
214212
|`TOPIC_RECREATE_DELAY_SECONDS` |Time delay between topic deletion and topic creation attempts for topic recreate functionality. Default: 1
215213
|`TOPIC_RECREATE_MAXRETRIES` |Number of attempts of topic creation after topic deletion for topic recreate functionality. Default: 15
214+
|`DYNAMIC_CONFIG_ENABLED`|Allow to change application config in runtime. Default: false.

‎documentation/compose/jaas/client.properties‎

100644100755
File mode changed.

‎documentation/compose/jaas/kafka_connect.jaas‎

100644100755
File mode changed.

‎documentation/compose/jaas/kafka_connect.password‎

100644100755
File mode changed.

‎documentation/compose/jaas/kafka_server.conf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,4 @@ KafkaClient {
1111
user_admin="admin-secret";
1212
};
1313

14-
Client {};
14+
Client {};

‎documentation/compose/jaas/schema_registry.jaas‎

100644100755
File mode changed.

‎documentation/compose/jaas/schema_registry.password‎

100644100755
File mode changed.

‎documentation/compose/kafka-ssl-components.yaml‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -15,27 +15,25 @@ services:
1515
KAFKA_CLUSTERS_0_NAME: local
1616
KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
1717
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka0:29092 # SSL LISTENER!
18-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /kafka.truststore.jks
19-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: secret
20-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /kafka.keystore.jks
21-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: secret
2218
KAFKA_CLUSTERS_0_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: '' # DISABLE COMMON NAME VERIFICATION
19+
2320
KAFKA_CLUSTERS_0_SCHEMAREGISTRY: https://schemaregistry0:8085
2421
KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTORELOCATION: /kafka.keystore.jks
2522
KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_KEYSTOREPASSWORD: "secret"
26-
KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTORELOCATION: /kafka.truststore.jks
27-
KAFKA_CLUSTERS_0_SCHEMAREGISTRYSSL_TRUSTSTOREPASSWORD: "secret"
23+
2824
KAFKA_CLUSTERS_0_KSQLDBSERVER: https://ksqldb0:8088
2925
KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTORELOCATION: /kafka.keystore.jks
3026
KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_KEYSTOREPASSWORD: "secret"
31-
KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTORELOCATION: /kafka.truststore.jks
32-
KAFKA_CLUSTERS_0_KSQLDBSERVERSSL_TRUSTSTOREPASSWORD: "secret"
27+
3328
KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: local
3429
KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: https://kafka-connect0:8083
3530
KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTORELOCATION: /kafka.keystore.jks
3631
KAFKA_CLUSTERS_0_KAFKACONNECT_0_KEYSTOREPASSWORD: "secret"
37-
KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTORELOCATION: /kafka.truststore.jks
38-
KAFKA_CLUSTERS_0_KAFKACONNECT_0_TRUSTSTOREPASSWORD: "secret"
32+
33+
KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION: /kafka.truststore.jks
34+
KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD: "secret"
35+
DYNAMIC_CONFIG_ENABLED: 'true' # not necessary for ssl, added for tests
36+
3937
volumes:
4038
- ./ssl/kafka.truststore.jks:/kafka.truststore.jks
4139
- ./ssl/kafka.keystore.jks:/kafka.keystore.jks

‎documentation/compose/kafka-ssl.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,11 @@ services:
1111
environment:
1212
KAFKA_CLUSTERS_0_NAME: local
1313
KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
14-
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092 # SSL LISTENER!
15-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /kafka.truststore.jks
16-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: secret
1714
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /kafka.keystore.jks
18-
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: secret
15+
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD: "secret"
16+
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092 # SSL LISTENER!
17+
KAFKA_CLUSTERS_0_SSL_TRUSTSTORELOCATION: /kafka.truststore.jks
18+
KAFKA_CLUSTERS_0_SSL_TRUSTSTOREPASSWORD: "secret"
1919
KAFKA_CLUSTERS_0_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: '' # DISABLE COMMON NAME VERIFICATION
2020
volumes:
2121
- ./ssl/kafka.truststore.jks:/kafka.truststore.jks
@@ -60,4 +60,4 @@ services:
6060
- ./ssl/creds:/etc/kafka/secrets/creds
6161
- ./ssl/kafka.truststore.jks:/etc/kafka/secrets/kafka.truststore.jks
6262
- ./ssl/kafka.keystore.jks:/etc/kafka/secrets/kafka.keystore.jks
63-
command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"
63+
command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"

‎documentation/compose/kafka-ui-arm64.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ services:
1919
KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry0:8085
2020
KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME: first
2121
KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS: http://kafka-connect0:8083
22+
DYNAMIC_CONFIG_ENABLED: 'true' # not necessary, added for tests
2223

2324
kafka0:
2425
image: confluentinc/cp-kafka:7.2.1.arm64

0 commit comments

Comments
 (0)