Skip to content

Commit 360b429

Browse files
committed
address comments
1 parent 026c529 commit 360b429

File tree

1 file changed

+15
-8
lines changed

1 file changed

+15
-8
lines changed

articles/hdinsight/spark/apache-spark-manage-jar-dependency.md renamed to articles/hdinsight/spark/manage-jar-dependency.md

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
2-
title: Manage Jar dependencies best practice for Azure HDInsight
3-
description: Best practice of how to manage Jar dependencies for HDInsight applications safely.
2+
title: Manage JAR dependencies - Azure HDInsight
3+
description: This article discusses best practices for managing Java Archive (JAR) dependencies for HDInsight applications.
44
author: hrasheed-msft
55
ms.author: hrasheed
66
ms.reviewer: jasonh
@@ -10,24 +10,24 @@ ms.topic: conceptual
1010
ms.date: 02/05/2020
1111
---
1212

13-
# Jar dependency management best practice
13+
# JAR dependency management best practices
1414

15-
Components in HDInsight cluster have dependencies on third-party libraries. Usually, a specific version of common modules like Guava is referenced by these built-in components. When customers submit an application with its dependencies, they can bring a conflict version of the same module. If customer’s version is in the classpath first, built-in components may throw exceptions because of version incompatibility. While if built-in components inject the dependency to classpath first, customer’s application may throw errors like NoSuchMethod.
15+
Components installed on HDInsight clusters have dependencies on third-party libraries. Usually, a specific version of common modules like Guava is referenced by these built-in components. When you submit an application with its dependencies, it can cause a conflict between different versions of the same module. If the component version that you reference in the classpath first, built-in components may throw exceptions because of version incompatibility. However, if built-in components inject their dependencies to the classpath first, your application may throw errors like `NoSuchMethod`.
1616

1717
To avoid version conflict, consider shading your application dependencies.
1818

1919
## What does package shading mean?
2020
Shading provides a way to include and rename dependencies. It relocates the classes and rewrites affected bytecode and resources to create a private copy of your dependencies.
2121

22-
## How to shade package?
22+
## How to shade a package?
2323

2424
### Use uber-jar
25-
Uber-jar is a single jar file that contains both the application jar and its dependencies. The dependencies in Uber-jar are by-default not shaded. In some cases, this may introduce version conflict if other components or applications reference a different version of those libraries. To avoid this, customer can build Uber-Jar file with some (or all) of their dependencies shaded.
25+
Uber-jar is a single jar file that contains both the application jar and its dependencies. The dependencies in Uber-jar are by-default not shaded. In some cases, this may introduce version conflict if other components or applications reference a different version of those libraries. To avoid this, you can build an Uber-Jar file with some (or all) of the dependencies shaded.
2626

2727
### Shade package using Maven
2828
Maven can build applications written both in Java and Scala. Maven-shade-plugin can help you create a shaded uber-jar easily.
2929

30-
Below example in `pom.xml` shows how to shade package using maven-shade-plugin. `<relocation>…</relocation>` moves classes from package `com.google.guava` into package `com.google.shaded.guava` by moving the corresponding JAR file entries and rewriting the affected bytecode.
30+
The example below shows a file `pom.xml` which has been updated to shade a package using maven-shade-plugin. The XML section `<relocation>…</relocation>` moves classes from package `com.google.guava` into package `com.google.shaded.guava` by moving the corresponding JAR file entries and rewriting the affected bytecode.
3131

3232
After changing `pom.xml`, you can execute `mvn package` to build the shaded uber-jar.
3333

@@ -61,7 +61,8 @@ After changing `pom.xml`, you can execute `mvn package` to build the shaded uber
6161

6262
### Shade package using SBT
6363
SBT is also a build tool for Scala and Java. SBT doesn't have a shade plugin like maven-shade-plugin. You can modify `build.sbt` file to shade packages.
64-
For example, to shade `com.google.guava`, you can add below command to `build.sbt` file:
64+
65+
For example, to shade `com.google.guava`, you can add the below command to the `build.sbt` file:
6566

6667
```scala
6768
assemblyShadeRules in assembly := Seq(
@@ -70,3 +71,9 @@ assemblyShadeRules in assembly := Seq(
7071
```
7172

7273
Then you can run `sbt clean` and `sbt assembly` to build the shaded jar file.
74+
75+
## Next steps
76+
77+
* [Use HDInsight IntelliJ Tools](https://docs.microsoft.com/azure/hdinsight/hadoop/hdinsight-tools-for-intellij-with-hortonworks-sandbox)
78+
79+
* [Create a Scala Maven application for Spark in IntelliJ](https://docs.microsoft.com/azure/hdinsight/spark/apache-spark-create-standalone-application)

0 commit comments

Comments
 (0)