Skip to content

Commit 5034a13

Browse files
Merge pull request #259361 from v-akarnase/patch-16
Update secure-spark-kafka-streaming-integration-scenario.md
2 parents b0a8418 + 96100d4 commit 5034a13

File tree

1 file changed

+10
-10
lines changed

1 file changed

+10
-10
lines changed

articles/hdinsight/kafka/secure-spark-kafka-streaming-integration-scenario.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,12 @@ ms.service: hdinsight
55
ms.topic: how-to
66
ms.author: piyushgupta
77
author: piyush-gupta1999
8-
ms.date: 11/03/2022
8+
ms.date: 11/23/2023
99
---
1010

1111
# Secure Spark and Kafka – Spark streaming integration scenario
1212

13-
In this document, you'll learn how to execute a Spark job in a secure Spark cluster that reads from a topic in secure Kafka cluster, provided the virtual networks are same/peered.
13+
In this document, you learn how to execute a Spark job in a secure Spark cluster that reads from a topic in secure Kafka cluster, provided the virtual networks are same/peered.
1414

1515
**Pre-requisites**
1616

@@ -64,7 +64,7 @@ In the Kafka cluster, set up Ranger policies and produce data from Kafka cluster
6464

6565
1. Add a Ranger policy for `bobadmin` with all accesses to all topics with wildcard pattern `*`
6666

67-
1. Execute the commands below based on your parameter values
67+
1. Execute the following commands based on your parameter values
6868

6969
```
7070
sshuser@hn0-umasec:~$ sudo apt -y install jq
@@ -141,7 +141,7 @@ In the Spark cluster, add entries in `/etc/hosts` in spark worker nodes, for Kaf
141141
142142
1. Create a keytab for user `alicetest` using ktutil tool. Let's call this file `alicetest.keytab`
143143
144-
1. Create a `bobadmin_jaas.conf` as shown in below sample
144+
1. Create a `bobadmin_jaas.conf` as shown in following sample
145145
146146
```
147147
KafkaClient {
@@ -154,7 +154,7 @@ In the Spark cluster, add entries in `/etc/hosts` in spark worker nodes, for Kaf
154154
principal="[email protected]";
155155
};
156156
```
157-
1. Create an `alicetest_jaas.conf` as shown in below sample
157+
1. Create an `alicetest_jaas.conf` as shown in following sample
158158
```
159159
KafkaClient {
160160
com.sun.security.auth.module.Krb5LoginModule required
@@ -207,7 +207,7 @@ From Spark cluster, read from kafka topic `alicetopic2` as user `alicetest` is a
207207
sshuser@hn0-umaspa:~$ spark-submit --num-executors 1 --master yarn --deploy-mode cluster --packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.2.3.1.0.4-1 --repositories http://repo.hortonworks.com/content/repositories/releases/ --files alicetest_jaas.conf#alicetest_jaas.conf,alicetest.keytab#alicetest.keytab --driver-java-options "-Djava.security.auth.login.config=./alicetest_jaas.conf" --class com.cloudera.spark.examples.DirectKafkaWordCount --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./alicetest_jaas.conf" /home/sshuser/spark-secure-kafka-app/target/spark-secure-kafka-app-1.0-SNAPSHOT.jar 10.3.16.118:9092 alicetopic2 false
208208
```
209209
210-
If you see the below error, which denotes the DNS (Domain Name Server) issue. Make sure to check Kafka worker nodes entry in `/etc/hosts` file in Spark cluster.
210+
If you see the following error, which denotes the DNS (Domain Name Server) issue. Make sure to check Kafka worker nodes entry in `/etc/hosts` file in Spark cluster.
211211
212212
```
213213
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))
@@ -219,13 +219,13 @@ From Spark cluster, read from kafka topic `alicetopic2` as user `alicetest` is a
219219
220220
1. From YARN UI, access the YARN job output you can see the `alicetest` user is able to read from `alicetopic2`. You can see the word count in the output.
221221
222-
1. Below are the detailed steps on how to check the application output from YARN UI.
222+
1. Following are the detailed steps on how to check the application output from YARN UI.
223223
224-
1. Go to YARN UI and open your application. Wait for the job to go to RUNNING state. You'll see the application details as below.
224+
1. Go to YARN UI and open your application. Wait for the job to go to RUNNING state. You'll see the following application details.
225225
226-
1. Click on Logs. You'll see the list of logs as shown below.
226+
1. Click Logs. You'll see the following list of logs.
227227
228-
1. Click on 'stdout'. You'll see the output with the count of words from your Kafka topic.
228+
1. Click 'stdout'. You'll see the following output with the count of words from your Kafka topic.
229229
230230
1. On the Kafka cluster’s Ranger UI, audit logs for the same will be shown.
231231

0 commit comments

Comments
 (0)