|
| 1 | += Security |
| 2 | + |
| 3 | +== Authentication |
| 4 | + |
| 5 | +Currently, the only supported authentication mechanism is Kerberos, which is disabled by default. |
| 6 | + |
| 7 | +Kerberos is a network authentication protocol that works on the basis of "tickets" to allow nodes communicating over a non-secure network to prove their identity to one another securely. It is used in Spark to authenticate users and to secure communication between Spark components. |
| 8 | + |
| 9 | +In this guide we show how to configure Spark applications to use Kerberos while accessing data in an HDFS cluster. The Stackable Secret Operator is used to generate the keytab files. In production environments, users might have different means to provision the keytab files. |
| 10 | + |
| 11 | + |
| 12 | +== Prerequisites |
| 13 | + |
| 14 | +It is assumed that you have a KDC server running in your cluster and that the Stackable Secret Operator is configured to provision the keytab files as described in the xref:home:secret-operator:secretclass.adoc#backend-kerberoskeytab[secret-operator documentation]. |
| 15 | + |
| 16 | +For details on HDFS and Kerberos, see the xref:hdfs-operator:usage-guide:security.adoc[HDFS operator guide]. |
| 17 | + |
| 18 | +This guide makes use of a SecretClass named `kerberos`. It is assumed that this class exists and is configured with a `kerberosBackend`. |
| 19 | + |
| 20 | +== Steps |
| 21 | + |
| 22 | +There are three steps to configure a Spark application to use Kerberos: |
| 23 | + |
| 24 | +1. Provision the Spark `driver` end `executor` pods with the keytab and `krb5.conf` files. |
| 25 | +2. Provision the Spark `job` pod with the keytab and `krb5.conf` files. |
| 26 | +3. Instruct the Spark application to use Kerberos. |
| 27 | + |
| 28 | +=== Driver and Executor pods |
| 29 | + |
| 30 | +Install the keytab and the `krb5.conf` files in the Spark pods. The keytab file contains the credentials of the user that is used to authenticate with the Kerberos server. The `krb5.conf` file contains the configuration settings for the Kerberos client. |
| 31 | + |
| 32 | +In the example below, the Stackable Secret Operator is used to provision the keytab via a volume claim. For brevity the configuration shared by the job, driver and executor pods is only specified once and then referenced in all other places where needed. |
| 33 | + |
| 34 | +[source,yaml] |
| 35 | +---- |
| 36 | +... |
| 37 | +job: |
| 38 | + config: &config |
| 39 | + volumeMounts: |
| 40 | + - name: kerberos |
| 41 | + mountPath: /stackable/kerberos <1> |
| 42 | + - name: kerberos |
| 43 | + mountPath: /etc/krb5.conf <2> |
| 44 | + subPath: krb5.conf |
| 45 | + - name: hdfs-config |
| 46 | + mountPath: /stackable/config/hdfs <3> |
| 47 | + envOverrides: |
| 48 | + HADOOP_CONF_DIR: /stackable/config/hdfs |
| 49 | +
|
| 50 | +driver: |
| 51 | + config: *config |
| 52 | +
|
| 53 | +executor: |
| 54 | + config: *config |
| 55 | +
|
| 56 | +volumes: |
| 57 | + - name: hdfs-config <4> |
| 58 | + configMap: |
| 59 | + name: hdfs |
| 60 | + - name: kerberos |
| 61 | + ephemeral: |
| 62 | + volumeClaimTemplate: |
| 63 | + metadata: |
| 64 | + annotations: |
| 65 | + secrets.stackable.tech/class: kerberos <5> |
| 66 | + secrets.stackable.tech/scope: service=spark <6> |
| 67 | + secrets.stackable.tech/kerberos.service.names: spark <7> |
| 68 | + spec: |
| 69 | + storageClassName: secrets.stackable.tech |
| 70 | + accessModes: |
| 71 | + - ReadWriteOnce |
| 72 | + resources: |
| 73 | + requests: |
| 74 | + storage: "1" |
| 75 | +---- |
| 76 | +<1> Mount the keytab from the kerberos volume. |
| 77 | +<2> Mount the `krb5.conf` file from the kerberos volume. |
| 78 | +<3> Mount the Hadoop configuration files from the `hdfs-config` module. |
| 79 | +<4> Hadoop configuration files as published by the Hdfs operator. |
| 80 | +<5> Name of the Secret class used to provision the keytab. |
| 81 | +<6> Scope of the Secret. |
| 82 | +<7> Name of the user for which the keytab is provisioned. |
| 83 | + |
| 84 | + |
| 85 | +=== Spark application |
| 86 | + |
| 87 | +Instruct the Spark application to use Kerberos by setting the `spark.kerberos.keytab` and `spark.kerberos.principal` properties in the `SparkApplication` CRD. |
| 88 | + |
| 89 | +[source,yaml] |
| 90 | +---- |
| 91 | +sparkConf: |
| 92 | + "spark.kerberos.keytab": "/stackable/kerberos/keytab" <1> |
| 93 | + "spark.kerberos.principal": "spark/[email protected]" <2> |
| 94 | +---- |
| 95 | +<1> Location of the keytab file. |
| 96 | +<2> Principal name. This needs to have the format `<SERVICE_NAME>.default.svc.cluster.local@<REALM>` where `SERVICE_NAME` matches the volume claim annotation `secrets.stackable.tech/kerberos.service.names` and `REALM` must be `CLUSTER.LOCAL` unless a different realm was used explicitly. In that case, the `KERBEROS_REALM` environment variable must also be set accordingly. |
| 97 | + |
0 commit comments