Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Basic Secure HDFS Support [514] #540

Merged
merged 16 commits into from
Dec 12, 2017
Merged

Basic Secure HDFS Support [514] #540

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
7612bf5
first stage of PR #514 of just logic
ifilonenko Nov 3, 2017
50f47d0
fixing build and unit test issues
ifilonenko Nov 3, 2017
87df4a7
fixed integration tests
ifilonenko Nov 3, 2017
67856a5
fixed issue with executorPodFactory unit tests
ifilonenko Nov 3, 2017
7cdae31
first series of PR comments
ifilonenko Nov 3, 2017
04aa26f
handle most PR comments
ifilonenko Nov 4, 2017
765455d
third round of PR comments
ifilonenko Nov 6, 2017
488b37e
initial round of comments and initial unit tests for deploy
ifilonenko Nov 25, 2017
37feb22
handled most of the comments and added test cases for pods
ifilonenko Nov 26, 2017
4e44027
resolve conflicts
ifilonenko Nov 29, 2017
86c7b8f
Merge branch 'branch-2.2-kubernetes' into hdfs-kerberos-support-1
ifilonenko Nov 29, 2017
64b0af7
merge conflicts
ifilonenko Nov 29, 2017
ba2bafc
adding thread sleeping for RSS issues as a test
ifilonenko Nov 29, 2017
0c99503
resolving comments and unit testing
ifilonenko Nov 29, 2017
a9d074b
regarding comments on PR
ifilonenko Dec 3, 2017
a3b12a7
merge conflicts and resolving comments
ifilonenko Dec 6, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...gers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/HadoopConfBootstrap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ private[spark] trait HadoopConfBootstrap {
private[spark] class HadoopConfBootstrapImpl(
hadoopConfConfigMapName: String,
hadoopConfigFiles: Seq[File],
hadoopUGI: HadoopUGIUtil) extends HadoopConfBootstrap with Logging{
hadoopUGI: HadoopUGIUtil) extends HadoopConfBootstrap with Logging {

override def bootstrapMainContainerAndVolumes(
originalPodWithMainContainer: PodWithMainContainer)
Expand Down
4 changes: 3 additions & 1 deletion ...e-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/HadoopUGIUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,9 @@ private[spark] class HadoopUGIUtil{
@deprecated("Moved to core in 2.3", "2.3")
def deserialize(tokenBytes: Array[Byte]): Credentials = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. Move to some other class?

val creds = new Credentials()
creds.readTokenStorageStream(new DataInputStream(new ByteArrayInputStream(tokenBytes)))
val dataStream = new DataInputStream(new ByteArrayInputStream(tokenBytes))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Utils.tryWithResource.

creds.readTokenStorageStream(dataStream)
dataStream.close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wraps close in a finally block.

creds
}
}
2 changes: 1 addition & 1 deletion ...bernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KerberosTokenConfBootstrap.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ private[spark] trait KerberosTokenConfBootstrap {
private[spark] class KerberosTokenConfBootstrapImpl(
secretName: String,
secretItemKey: String,
userName: String) extends KerberosTokenConfBootstrap with Logging{
userName: String) extends KerberosTokenConfBootstrap with Logging {

override def bootstrapMainContainerAndVolumes(
originalPodWithMainContainer: PodWithMainContainer)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation is off here, indent in one more along with the line below.

Expand Down
4 changes: 2 additions & 2 deletions ...main/scala/org/apache/spark/deploy/k8s/submit/submitsteps/HadoopConfigBootstrapStep.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,13 @@ private[spark] class HadoopConfigBootstrapStep(
.endMetadata()
.addToData(currentHadoopSpec.configMapProperties.asJava)
.build()
val executorSparkConf = driverSpec.driverSparkConf.clone()
val driverSparkConfWithExecutorSetup = driverSpec.driverSparkConf.clone()
.set(HADOOP_CONFIG_MAP_SPARK_CONF_NAME, hadoopConfigMapName)
.setAll(currentHadoopSpec.additionalDriverSparkConf)
driverSpec.copy(
driverPod = currentHadoopSpec.driverPod,
driverContainer = currentHadoopSpec.driverContainer,
driverSparkConf = executorSparkConf,
driverSparkConf = driverSparkConfWithExecutorSetup,
otherKubernetesResources =
driverSpec.otherKubernetesResources ++
Seq(configMap) ++ currentHadoopSpec.dtSecret.toSeq
Expand Down