update

Author: razvan

docs/modules/spark-k8s/pages/usage-guide/security.adoc

@@ -6,16 +6,16 @@ Currently, the only supported authentication mechanism is Kerberos, which is dis
 
 Kerberos is a network authentication protocol that works on the basis of "tickets" to allow nodes communicating over a non-secure network to prove their identity to one another securely. It is used in Spark to authenticate users and to secure communication between Spark components.
 
-In this guide we show how to configure Spark applications to use Kerberos. The Stackable Secret Operator is used to generate the keytab files. In production environments, users might have different means to provision the keytab files.
+In this guide we show how to configure Spark applications to use Kerberos while accessing data in HDFS cluster. The Stackable Secret Operator is used to generate the keytab files. In production environments, users might have different means to provision the keytab files.
 
 
 == Prerequisites
 
 It is assumed that you have a KDC server running in your cluster and that the Stackable Secret Operator is configured to provision the keytab files as described in xref:home:secret-operator:secretclass.adoc#backend-kerberoskeytab[secret-operator documentation].
 
-If the Spark application processes data from a kerberized Hadoop cluster, follow the xref:hdfs-operator:usage-guide:security.adoc[HDFS operator guide] to configure HDFS with Kerberos.
+For details on HDFS and Kerberos, see the xref:hdfs-operator:usage-guide:security.adoc[HDFS operator guide].
 
-This guide makes use of a SecretClass named `kerberos-default`. It is assumed that this class exists and is configured with a `kerberosBackend`.
+This guide makes use of a SecretClass named `kerberos`. It is assumed that this class exists and is configured with a `kerberosBackend`.
 
 == Steps
 
@@ -29,33 +29,42 @@ There are three steps to configure a Spark application to use Kerberos:
 
 Install the keytab and the `krb5.conf` files in the Spark pods. The keytab file contains the credentials of the user that is used to authenticate with the Kerberos server. The `krb5.conf` file contains the configuration settings for the Kerberos client.
 
-In the example below, the Stackable Secret Operator is used to provision the keytab via a volume claim. The `krb5.conf` file is mounted as a ConfigMap.
+In the example below, the Stackable Secret Operator is used to provision the keytab via a volume claim. For brevity the configuration shared by the job, driver and executor pods is only specified once and then referenced in all other places where needed.
 
 [source,yaml]
 ----
 ...
-driver:
-  config:
+job:
+  config: &config
     volumeMounts:
       - name: kerberos
         mountPath: /stackable/kerberos <1>
-executor:
-  config:
-    volumeMounts:
       - name: kerberos
-        mountPath: /stackable/kerberos <2>
+        mountPath: /etc/krb5.conf <2>
+        subPath: krb5.conf
+      - name: hdfs-config
+        mountPath: /stackable/config/hdfs <3>
+    envOverrides:
+      HADOOP_CONF_DIR: /stackable/config/hdfs
+
+driver:
+  config: *config
+
+executor:
+  config: *config
+
 volumes:
-  - name: kerberos-config
+  - name: hdfs-config <4>
     configMap:
-      name: krb5-kdc <3>
+      name: hdfs 
   - name: kerberos
     ephemeral:
       volumeClaimTemplate:
         metadata:
           annotations:
-            secrets.stackable.tech/class: kerberos-default <4>
-            secrets.stackable.tech/scope: service=spark-teragen <5>
-            secrets.stackable.tech/kerberos.service.names: testuser <6>
+            secrets.stackable.tech/class: kerberos <5>
+            secrets.stackable.tech/scope: service=spark <6>
+            secrets.stackable.tech/kerberos.service.names: spark <7>
         spec:
           storageClassName: secrets.stackable.tech
           accessModes:
@@ -64,100 +73,25 @@ volumes:
             requests:
               storage: "1"
 ----
-<1> Mount the keytab volume in the driver pod.
-<2> Mount the keytab volume in the executor pods.
-<3> Mount the `krb5.conf` file as a ConfigMap.
-<4> Name of the Secret class used to provision the keytab.
-<5> Scope of the Secret.
-<6> Name of the user for which the keytab is provisioned.
-
-
-=== Job pod
-
-Install the keytab and the `krb5.conf` files in the Spark `job` pod. This must be currently done via pod overrides. This is because the Spark application volumes are not currently visible to the `job` pod. We hope to address this limitation in a future release.
-
-[source,yaml]
-----
-job:
-  podOverrides:
-    spec:
-      volumes:
-        - name: kerberos-config
-          configMap:
-            name: krb5-kdc <1>
-        - name: kerberos
-          ephemeral:
-            volumeClaimTemplate:
-              metadata:
-                annotations:
-                  secrets.stackable.tech/class: kerberos-default <2>
-                  secrets.stackable.tech/scope: service=spark-teragen <3>
-                  secrets.stackable.tech/kerberos.service.names: testuser <4>
-              spec:
-                storageClassName: secrets.stackable.tech
-                accessModes:
-                  - ReadWriteOnce
-                resources:
-                  requests:
-                    storage: "1"
-      containers:
-        - name: spark-submit
-          volumeMounts:
-            - name: kerberos <5>
-              mountPath: /stackable/kerberos
-----
-<1> Mount the `krb5.conf` file as a ConfigMap.
-<2> Name of the Secret class used to provision the keytab.
-<3> Scope of the Secret.
-<4> Name of the user for which the keytab is provisioned.
-<5> Mount the keytab volume in the job pod.
-
+<1> Mount the keytab from the kerberos volume.
+<2> Mount the `krb5.conf` file from the kerberos volume.
+<3> Mount the Hadoop configuration files from the `hdfs-config` module.
+<4> Hadoop configuration files as published by the Hdfs operator.
+<5> Name of the Secret class used to provision the keytab.
+<6> Scope of the Secret.
+<7> Name of the user for which the keytab is provisioned.
 
 
 === Spark application
 
 Instruct the Spark application to use Kerberos by setting the `spark.kerberos.keytab` and `spark.kerberos.principal` properties in the `SparkApplication` CRD.
 
-Finally, instruct Spark to use the keytab and `krb5.conf` files provisioned in the previous steps.
-
 [source,yaml]
 ----
 sparkConf:
   "spark.kerberos.keytab": "/stackable/kerberos/keytab" <1>
-  "spark.kerberos.principal": "testuser/[email protected]" <2>
-  "spark.driver.extraJavaOptions": "-Djava.security.krb5.conf=/stackable/kerberos/krb5.conf" <3>
-  "spark.executor.extraJavaOptions": "-Djava.security.krb5.conf=/stackable/kerberos/krb5.conf" <4>
+  "spark.kerberos.principal": "spark/[email protected]" <2>
 ----
 <1> Location of the keytab file.
-<2> Principal name. This needs to have the format `<SERVICE_NAME>.default.svc.cluster.local@<REALM>` where `SERVICE_NAME` matches the volume claim annotation `secrets.stackable.tech/kerberos.service.names` and `REALM` must be `CLUSTER.LOCAL`.
-<3> Location of the Kerberos configuration for the application driver.
-<4> Location of the Kerberos configuration for the application executors.
-
-=== Hadoop
-
-When reading and writing data from a kerberized Hadoop cluster, a the HDFS discovery map must mounted the `SparkApplication` pods as follows:
-
-For the driver and executor pods:
-
-[source,yaml]
-----
-...
-driver:
-  config:
-    volumeMounts:
-      - name: hdfs-config
-        mountPath: /etc/hadoop/conf <1>
-executor:
-  config:
-    volumeMounts:
-      - name: hdfs-config
-        mountPath: /etc/hadoop/conf <2>
-volumes:
-  - name: hdfs-config
-    configMap:
-      name: hdfs-discovery-cm <3>
-----
-<1> Location of the HDFS configuration for the driver.
-<2> Location of the HDFS configuration for the executors.
-<3> Name of the HDFS discovery ConfigMap as published by the HDFS operator.
+<2> Principal name. This needs to have the format `<SERVICE_NAME>.default.svc.cluster.local@<REALM>` where `SERVICE_NAME` matches the volume claim annotation `secrets.stackable.tech/kerberos.service.names` and `REALM` must be `CLUSTER.LOCAL` unless a different realm was used explicitly. In that case, the `KERBEROS_REALM` environment variable must also be set accordingly.