You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight/spark/manage-jar-dependency.md
+15-8Lines changed: 15 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
-
title: Manage Jar dependencies best practice for Azure HDInsight
3
-
description: Best practice of how to manage Jar dependencies for HDInsight applications safely.
2
+
title: Manage JAR dependencies - Azure HDInsight
3
+
description: This article discusses best practices for managing Java Archive (JAR) dependencies for HDInsight applications.
4
4
author: hrasheed-msft
5
5
ms.author: hrasheed
6
6
ms.reviewer: jasonh
@@ -10,24 +10,24 @@ ms.topic: conceptual
10
10
ms.date: 02/05/2020
11
11
---
12
12
13
-
# Jar dependency management best practice
13
+
# JAR dependency management best practices
14
14
15
-
Components in HDInsight cluster have dependencies on third-party libraries. Usually, a specific version of common modules like Guava is referenced by these built-in components. When customers submit an application with its dependencies, they can bring a conflict version of the same module. If customer’s version is in the classpath first, built-in components may throw exceptions because of version incompatibility. While if built-in components inject the dependency to classpath first, customer’s application may throw errors like NoSuchMethod.
15
+
Components installed on HDInsight clusters have dependencies on third-party libraries. Usually, a specific version of common modules like Guava is referenced by these built-in components. When you submit an application with its dependencies, it can cause a conflict between different versions of the same module. If the component version that you reference in the classpath first, built-in components may throw exceptions because of version incompatibility. However, if built-in components inject their dependencies to the classpath first, your application may throw errors like `NoSuchMethod`.
16
16
17
17
To avoid version conflict, consider shading your application dependencies.
18
18
19
19
## What does package shading mean?
20
20
Shading provides a way to include and rename dependencies. It relocates the classes and rewrites affected bytecode and resources to create a private copy of your dependencies.
21
21
22
-
## How to shade package?
22
+
## How to shade a package?
23
23
24
24
### Use uber-jar
25
-
Uber-jar is a single jar file that contains both the application jar and its dependencies. The dependencies in Uber-jar are by-default not shaded. In some cases, this may introduce version conflict if other components or applications reference a different version of those libraries. To avoid this, customer can build Uber-Jar file with some (or all) of their dependencies shaded.
25
+
Uber-jar is a single jar file that contains both the application jar and its dependencies. The dependencies in Uber-jar are by-default not shaded. In some cases, this may introduce version conflict if other components or applications reference a different version of those libraries. To avoid this, you can build an Uber-Jar file with some (or all) of the dependencies shaded.
26
26
27
27
### Shade package using Maven
28
28
Maven can build applications written both in Java and Scala. Maven-shade-plugin can help you create a shaded uber-jar easily.
29
29
30
-
Below example in `pom.xml`shows how to shade package using maven-shade-plugin. `<relocation>…</relocation>` moves classes from package `com.google.guava` into package `com.google.shaded.guava` by moving the corresponding JAR file entries and rewriting the affected bytecode.
30
+
The example below shows a file `pom.xml`which has been updated to shade a package using maven-shade-plugin. The XML section`<relocation>…</relocation>` moves classes from package `com.google.guava` into package `com.google.shaded.guava` by moving the corresponding JAR file entries and rewriting the affected bytecode.
31
31
32
32
After changing `pom.xml`, you can execute `mvn package` to build the shaded uber-jar.
33
33
@@ -61,7 +61,8 @@ After changing `pom.xml`, you can execute `mvn package` to build the shaded uber
61
61
62
62
### Shade package using SBT
63
63
SBT is also a build tool for Scala and Java. SBT doesn't have a shade plugin like maven-shade-plugin. You can modify `build.sbt` file to shade packages.
64
-
For example, to shade `com.google.guava`, you can add below command to `build.sbt` file:
64
+
65
+
For example, to shade `com.google.guava`, you can add the below command to the `build.sbt` file:
65
66
66
67
```scala
67
68
assemblyShadeRules in assembly :=Seq(
@@ -70,3 +71,9 @@ assemblyShadeRules in assembly := Seq(
70
71
```
71
72
72
73
Then you can run `sbt clean` and `sbt assembly` to build the shaded jar file.
0 commit comments