diff --git a/docker/examples/README.md b/docker/examples/README.md index 162e27c711aca..ac6efb5f20a41 100644 --- a/docker/examples/README.md +++ b/docker/examples/README.md @@ -73,6 +73,18 @@ It is also possible to use the input file to have a common set of configurations - Assuming that `KAFKA_LOG4J_LOGGERS='property1=value1,property2=value2'` environment variable is provided to Docker container. - `log4j.logger.property1=value1` and `log4j.logger.property2=value2` will be added to the `log4j2.yaml` file inside Docker container. +Running in SASL mode +-------------------- + +- SASL mode requires a JAAS configuration file to be mounted into the Docker container and referenced via the `KAFKA_OPTS` environment variable. +- Mount the directory containing the JAAS config file to `/etc/kafka/secrets` in the Docker container. +- Set `KAFKA_OPTS` to `-Djava.security.auth.login.config=/etc/kafka/secrets/`. +- Set `KAFKA_SASL_ENABLED_MECHANISMS` to the desired SASL mechanism (e.g. `GSSAPI` , `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`). +- Ensure `KAFKA_ADVERTISED_LISTENERS` contains a `SASL_PLAINTEXT://` or `SASL_SSL://` listener. +- For inter-broker SASL communication, set `KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL` to the desired mechanism (SASL_PLAINTEXT (or SASL_SSL)). +- The Docker image `configure` script will validate that `KAFKA_OPTS` contains the `java.security.auth.login.config` property when SASL listeners are detected. +- For more details on SASL mechanisms and configuration, refer to the [Kafka SASL authentication documentation](https://kafka.apache.org/documentation/#security_sasl). + Running in SSL mode ------------------- @@ -169,6 +181,25 @@ Single Node # Run from root of the repo $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties ``` +- SASL_PLAINTEXT: + - Here we use SASL/PLAIN mechanism for authentication. + - A JAAS configuration file is mounted into the container via a volume mount. + - `KAFKA_OPTS` is set to point to the JAAS config file. + - Similar to the plaintext example, two listeners are configured: one for inter-broker communication and one for client-to-broker communication. Both use the `SASL_PLAINTEXT` security protocol. + - Two users are configured in the JAAS file: `admin` (for inter-broker) and `alice` (for clients). + - Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details. + - To run the example: + ``` + # Run from root of the repo + + # JVM based Apache Kafka Docker Image + $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml up + ``` + - To produce messages using client scripts (Ensure that java version >= 17): + ``` + # Run from root of the repo + $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties + ``` Multi Node Cluster ------------------ @@ -221,6 +252,23 @@ Multi Node Cluster # Run from root of the repo $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties ``` + - SASL_PLAINTEXT: + - Similar to Plaintext example, with SASL/PLAIN authentication added. + - Similar to Plaintext example, two listeners are configured for inter-broker and client-to-broker communication, both using the `SASL_PLAINTEXT` security protocol. Controllers use `PLAINTEXT`. + - Each broker mounts the same JAAS config file. In production, each broker should have its own credentials. + - Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details. + - To run the example: + ``` + # Run from root of the repo + + # JVM based Apache Kafka Docker Image + $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml up + ``` + - To produce messages using client scripts (Ensure that java version >= 17): + ``` + # Run from root of the repo + $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties + ``` - Isolated: - Examples are present in `docker-compose-files/cluster/isolated` directory. - Plaintext: @@ -260,5 +308,21 @@ Multi Node Cluster # Run from root of the repo $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29093 --command-config ./docker/examples/fixtures/client-secrets/client-ssl.properties ``` + - SASL_PLAINTEXT: + - Same as combined SASL_PLAINTEXT example, with controllers and brokers separated. + - `SASL_PLAINTEXT` is only for inter-broker and client communication. Controllers use `PLAINTEXT`. + - Note: SASL is currently not supported with the GraalVM based native image (`apache/kafka-native`) due to missing reflection configuration for `java.security.AccessController`. See [KAFKA-19584](https://issues.apache.org/jira/browse/KAFKA-19584) for details. + - To run the example: + ``` + # Run from root of the repo + + # JVM based Apache Kafka Docker Image + $ IMAGE=apache/kafka:latest docker compose -f docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml up + ``` + - To produce messages using client scripts (Ensure that java version >= 17): + ``` + # Run from root of the repo + $ bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:29094 --command-config ./docker/examples/fixtures/client-secrets/client-sasl.properties + ``` - Note that the examples are meant to be tried one at a time, make sure you close an example server before trying out the other to avoid conflicts. diff --git a/docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml b/docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml new file mode 100644 index 0000000000000..17fadb498914b --- /dev/null +++ b/docker/examples/docker-compose-files/cluster/combined/sasl_plaintext/docker-compose.yml @@ -0,0 +1,103 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- +services: + kafka-1: + image: ${IMAGE} + hostname: kafka-1 + container_name: kafka-1 + ports: + - 29094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: 'broker,controller' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-1:19094,SASL_PLAINTEXT_HOST://localhost:29094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' + + kafka-2: + image: ${IMAGE} + hostname: kafka-2 + container_name: kafka-2 + ports: + - 39094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 2 + KAFKA_PROCESS_ROLES: 'broker,controller' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-2:19094,SASL_PLAINTEXT_HOST://localhost:39094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' + + kafka-3: + image: ${IMAGE} + hostname: kafka-3 + container_name: kafka-3 + ports: + - 49094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 3 + KAFKA_PROCESS_ROLES: 'broker,controller' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,CONTROLLER://:9093,SASL_PLAINTEXT_HOST://:9094' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-3:19094,SASL_PLAINTEXT_HOST://localhost:49094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' diff --git a/docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml b/docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml new file mode 100644 index 0000000000000..a86379a6e31e6 --- /dev/null +++ b/docker/examples/docker-compose-files/cluster/isolated/sasl_plaintext/docker-compose.yml @@ -0,0 +1,166 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- +services: + controller-1: + image: ${IMAGE} + environment: + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: 'controller' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + KAFKA_LISTENERS: 'CONTROLLER://:9093' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + + controller-2: + image: ${IMAGE} + environment: + KAFKA_NODE_ID: 2 + KAFKA_PROCESS_ROLES: 'controller' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + KAFKA_LISTENERS: 'CONTROLLER://:9093' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + + controller-3: + image: ${IMAGE} + environment: + KAFKA_NODE_ID: 3 + KAFKA_PROCESS_ROLES: 'controller' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + KAFKA_LISTENERS: 'CONTROLLER://:9093' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + + kafka-1: + image: ${IMAGE} + hostname: kafka-1 + container_name: kafka-1 + ports: + - 29094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 4 + KAFKA_PROCESS_ROLES: 'broker' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-1:19094,SASL_PLAINTEXT_HOST://localhost:29094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' + depends_on: + - controller-1 + - controller-2 + - controller-3 + + kafka-2: + image: ${IMAGE} + hostname: kafka-2 + container_name: kafka-2 + ports: + - 39094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 5 + KAFKA_PROCESS_ROLES: 'broker' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-2:19094,SASL_PLAINTEXT_HOST://localhost:39094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' + depends_on: + - controller-1 + - controller-2 + - controller-3 + + kafka-3: + image: ${IMAGE} + hostname: kafka-3 + container_name: kafka-3 + ports: + - 49094:9094 + volumes: + - ../../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 6 + KAFKA_PROCESS_ROLES: 'broker' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_LISTENERS: 'SASL_PLAINTEXT://:19094,SASL_PLAINTEXT_HOST://:9094' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller-1:9093,2@controller-2:9093,3@controller-3:9093' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://kafka-3:19094,SASL_PLAINTEXT_HOST://localhost:49094 + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' + depends_on: + - controller-1 + - controller-2 + - controller-3 diff --git a/docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml b/docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml new file mode 100644 index 0000000000000..082e9fe98f36d --- /dev/null +++ b/docker/examples/docker-compose-files/single-node/sasl_plaintext/docker-compose.yml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- +services: + broker: + image: ${IMAGE} + hostname: broker + container_name: broker + ports: + - '9094:9094' + volumes: + - ../../../fixtures/sasl:/etc/kafka/secrets + environment: + KAFKA_NODE_ID: 1 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_PLAINTEXT_HOST:SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: 'SASL_PLAINTEXT_HOST://localhost:9094,SASL_PLAINTEXT://broker:19094' + KAFKA_PROCESS_ROLES: 'broker,controller' + KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093' + KAFKA_LISTENERS: 'CONTROLLER://:29093,SASL_PLAINTEXT_HOST://:9094,SASL_PLAINTEXT://:19094' + KAFKA_INTER_BROKER_LISTENER_NAME: 'SASL_PLAINTEXT' + KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' + CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw' + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 + KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/secrets/broker_jaas.conf' diff --git a/docker/examples/fixtures/client-secrets/client-sasl.properties b/docker/examples/fixtures/client-secrets/client-sasl.properties new file mode 100644 index 0000000000000..710d693078463 --- /dev/null +++ b/docker/examples/fixtures/client-secrets/client-sasl.properties @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +security.protocol=SASL_PLAINTEXT +sasl.mechanism=PLAIN +sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ + username="alice" \ + password="alice-secret"; diff --git a/docker/examples/fixtures/sasl/broker_jaas.conf b/docker/examples/fixtures/sasl/broker_jaas.conf new file mode 100644 index 0000000000000..d2faf584730ce --- /dev/null +++ b/docker/examples/fixtures/sasl/broker_jaas.conf @@ -0,0 +1,22 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +KafkaServer { + org.apache.kafka.common.security.plain.PlainLoginModule required + username="admin" + password="admin-secret" + user_admin="admin-secret" + user_alice="alice-secret"; +}; diff --git a/docker/test/constants.py b/docker/test/constants.py index 710e29b69611d..5bc480b36c273 100644 --- a/docker/test/constants.py +++ b/docker/test/constants.py @@ -30,6 +30,10 @@ FILE_INPUT_FLOW_TESTS="File Input Flow Tests" FILE_INPUT_TOPIC="test-topic-file-input" +SASL_FLOW_TESTS="SASL Flow Tests" +SASL_CLIENT_CONFIG="fixtures/sasl/client-sasl.properties" +SASL_TOPIC="test-topic-sasl" + BROKER_RESTART_TESTS="Broker Restart Tests" BROKER_CONTAINER="broker1" BROKER_RESTART_TEST_TOPIC="test-topic-broker-restart" @@ -40,6 +44,7 @@ BROKER_METRICS_HEADING='"time","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:Count","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:EventType","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:MeanRate","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:OneMinuteRate","kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:RateUnit"' SSL_ERROR_PREFIX="SSL_ERR" +SASL_ERROR_PREFIX="SASL_ERR" BROKER_RESTART_ERROR_PREFIX="BROKER_RESTART_ERR" FILE_INPUT_ERROR_PREFIX="FILE_INPUT_ERR" BROKER_METRICS_ERROR_PREFIX="BROKER_METRICS_ERR" diff --git a/docker/test/docker_sanity_test.py b/docker/test/docker_sanity_test.py index d6e648cbe0bf1..1cf2a5e08f0d4 100644 --- a/docker/test/docker_sanity_test.py +++ b/docker/test/docker_sanity_test.py @@ -24,6 +24,7 @@ class DockerSanityTest(unittest.TestCase): IMAGE="apache/kafka" FIXTURES_DIR="." + MODE="jvm" def resume_container(self): subprocess.run(["docker", "start", constants.BROKER_CONTAINER]) @@ -145,6 +146,32 @@ def ssl_flow(self, ssl_broker_port, test_name, test_error_prefix, topic): return errors + def sasl_flow(self, sasl_broker_port, test_name, test_error_prefix, topic): + print(f"Running {test_name}") + errors = [] + try: + self.assertTrue(self.create_topic(topic, ["--bootstrap-server", sasl_broker_port, "--command-config", f"{self.FIXTURES_DIR}/{constants.SASL_CLIENT_CONFIG}"])) + except AssertionError as e: + errors.append(test_error_prefix + str(e)) + return errors + + producer_config = ["--bootstrap-server", sasl_broker_port, + "--command-config", f"{self.FIXTURES_DIR}/{constants.SASL_CLIENT_CONFIG}"] + self.produce_message(topic, producer_config, "key", "message") + + consumer_config = [ + "--bootstrap-server", sasl_broker_port, + "--command-property", "auto.offset.reset=earliest", + "--command-config", f"{self.FIXTURES_DIR}/{constants.SASL_CLIENT_CONFIG}", + ] + message = self.consume_message(topic, consumer_config) + try: + self.assertEqual(message, "key:message") + except AssertionError as e: + errors.append(test_error_prefix + str(e)) + + return errors + def broker_restart_flow(self): print(f"Running {constants.BROKER_RESTART_TESTS}") errors = [] @@ -189,6 +216,13 @@ def execute(self): except Exception as e: print(constants.FILE_INPUT_ERROR_PREFIX, str(e)) total_errors.append(str(e)) + # SASL is not supported on native image due to missing reflection config (KAFKA-19584) + if self.MODE == "jvm": + try: + total_errors.extend(self.sasl_flow('localhost:9095', constants.SASL_FLOW_TESTS, constants.SASL_ERROR_PREFIX, constants.SASL_TOPIC)) + except Exception as e: + print(constants.SASL_ERROR_PREFIX, str(e)) + total_errors.append(str(e)) try: total_errors.extend(self.broker_restart_flow()) except Exception as e: @@ -216,6 +250,7 @@ def test_bed(self): def run_tests(image, mode, fixtures_dir): DockerSanityTest.IMAGE = image DockerSanityTest.FIXTURES_DIR = fixtures_dir + DockerSanityTest.MODE = mode test_classes_to_run = [] if mode == "jvm" or mode == "native": diff --git a/docker/test/fixtures/mode/combined/docker-compose.yml b/docker/test/fixtures/mode/combined/docker-compose.yml index 8691019d02447..28513f58a96aa 100644 --- a/docker/test/fixtures/mode/combined/docker-compose.yml +++ b/docker/test/fixtures/mode/combined/docker-compose.yml @@ -23,12 +23,14 @@ services: - "9092:9092" - "9101:9101" - "19093:9093" + - "9095:9095" volumes: - ../../secrets:/etc/kafka/secrets + - ../../sasl:/etc/kafka/sasl environment: KAFKA_NODE_ID: 1 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,PLAINTEXT_HOST:PLAINTEXT' - KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,SSL://localhost:19093,PLAINTEXT://broker1:29092' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,PLAINTEXT_HOST:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,SSL://localhost:19093,PLAINTEXT://broker1:29092,SASL_PLAINTEXT://localhost:9095' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 @@ -37,7 +39,7 @@ services: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker1:19092,2@broker2:19092,3@broker3:19092' - KAFKA_LISTENERS: 'CONTROLLER://:19092,PLAINTEXT_HOST://:9092,SSL://:9093,PLAINTEXT://:29092' + KAFKA_LISTENERS: 'CONTROLLER://:19092,PLAINTEXT_HOST://:9092,SSL://:9093,PLAINTEXT://:29092,SASL_PLAINTEXT://:9095' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' @@ -50,6 +52,8 @@ services: KAFKA_SSL_TRUSTSTORE_FILENAME: "kafka.truststore.jks" KAFKA_SSL_TRUSTSTORE_CREDENTIALS: "kafka_truststore_creds" KAFKA_SSL_CLIENT_AUTH: "required" + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/sasl/broker_jaas.conf' broker2: image: {$IMAGE} diff --git a/docker/test/fixtures/mode/isolated/docker-compose.yml b/docker/test/fixtures/mode/isolated/docker-compose.yml index 2c77fc123f137..24a05924f4a50 100644 --- a/docker/test/fixtures/mode/isolated/docker-compose.yml +++ b/docker/test/fixtures/mode/isolated/docker-compose.yml @@ -83,12 +83,14 @@ services: - "9092:9092" - "19093:9093" - "9101:9101" + - "9095:9095" volumes: - ../../secrets:/etc/kafka/secrets + - ../../sasl:/etc/kafka/sasl environment: KAFKA_NODE_ID: 4 - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,PLAINTEXT_HOST:PLAINTEXT' - KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,SSL://localhost:19093,PLAINTEXT://broker1:29092' + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,PLAINTEXT_HOST:PLAINTEXT,SASL_PLAINTEXT:SASL_PLAINTEXT' + KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://localhost:9092,SSL://localhost:19093,PLAINTEXT://broker1:29092,SASL_PLAINTEXT://localhost:9095' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 @@ -97,7 +99,7 @@ services: KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1 KAFKA_PROCESS_ROLES: 'broker' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@controller1:19092,2@controller2:19092,3@controller3:19092' - KAFKA_LISTENERS: 'PLAINTEXT_HOST://:9092,SSL://:9093,PLAINTEXT://:29092' + KAFKA_LISTENERS: 'PLAINTEXT_HOST://:9092,SSL://:9093,PLAINTEXT://:29092,SASL_PLAINTEXT://:9095' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs' @@ -110,6 +112,8 @@ services: KAFKA_SSL_TRUSTSTORE_FILENAME: "kafka.truststore.jks" KAFKA_SSL_TRUSTSTORE_CREDENTIALS: "kafka_truststore_creds" KAFKA_SSL_CLIENT_AUTH: "required" + KAFKA_SASL_ENABLED_MECHANISMS: 'PLAIN' + KAFKA_OPTS: '-Djava.security.auth.login.config=/etc/kafka/sasl/broker_jaas.conf' depends_on: - controller1 - controller2 diff --git a/docker/test/fixtures/sasl/broker_jaas.conf b/docker/test/fixtures/sasl/broker_jaas.conf new file mode 100644 index 0000000000000..d2faf584730ce --- /dev/null +++ b/docker/test/fixtures/sasl/broker_jaas.conf @@ -0,0 +1,22 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +KafkaServer { + org.apache.kafka.common.security.plain.PlainLoginModule required + username="admin" + password="admin-secret" + user_admin="admin-secret" + user_alice="alice-secret"; +}; diff --git a/docker/test/fixtures/sasl/client-sasl.properties b/docker/test/fixtures/sasl/client-sasl.properties new file mode 100644 index 0000000000000..710d693078463 --- /dev/null +++ b/docker/test/fixtures/sasl/client-sasl.properties @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +security.protocol=SASL_PLAINTEXT +sasl.mechanism=PLAIN +sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ + username="alice" \ + password="alice-secret";