Skip to content

Commit 2f9e2a1

Browse files
committed
[SPARK-53827] Exclude commons-(codec|compress) and com.ibm.icu transitive dependencies
### What changes were proposed in this pull request? This PR aims to exclude 3 transitive dependencies to reduce the size of `spark-kubernetes-operator.jar` about **11% (from 149MB to 133MB)**. - `commons-codec:*` - `org.apache.commons:commons-compress` - `com.ibm.icu:*` ### Why are the changes needed? Apache Spark K8s Operator is not using these. We had better remove this to reduce `spark-kubernetes-operator.jar` file size (and the Docker image size) from 149MB to 133MB. **BEFORE** ``` $ docker run -it --rm apache/spark-kubernetes-operator:0.6.0-SNAPSHOT ls -alh /opt/spark-operator/operator | tail -n1 / -rw-r--r-- 1 spark spark 149M Oct 7 23:21 spark-kubernetes-operator.jar ``` **AFTER** ``` $ docker run -it --rm apache/spark-kubernetes-operator:0.6.0-SNAPSHOT ls -alh /opt/spark-operator/operator | tail -n1 / -rw-r--r-- 1 spark spark 133M Oct 7 23:28 spark-kubernetes-operator.jar ``` ### Does this PR introduce _any_ user-facing change? No behavior change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#375 from dongjoon-hyun/SPARK-53827. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ae7c376 commit 2f9e2a1

File tree

3 files changed

+8
-0
lines changed

3 files changed

+8
-0
lines changed

config/checkstyle/checkstyle.xml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,8 +201,10 @@
201201
value="Please use the `assertThrows` method to test for exceptions."/>
202202
</module>
203203
<module name="IllegalImport">
204+
<property name="illegalPkgs" value="org.apache.commons.codec"/>
204205
<property name="illegalPkgs" value="org.apache.commons.collections"/>
205206
<property name="illegalPkgs" value="org.apache.commons.collections4" />
207+
<property name="illegalPkgs" value="org.apache.commons.compress" />
206208
<property name="illegalPkgs" value="org.apache.commons.lang"/>
207209
<property name="illegalPkgs" value="org.apache.commons.lang3" />
208210
<property name="illegalPkgs" value="org.apache.log4j"/>

spark-operator/build.gradle

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,12 @@ dependencies {
4040
implementation(libs.metrics.jvm)
4141
compileOnly(libs.spark.core) {
4242
exclude group: "com.github.luben"
43+
exclude group: "com.ibm.icu"
44+
exclude group: "commons-codec"
4345
exclude group: "commons-collections", module: "commons-collections"
4446
exclude group: "io.netty", module: "netty-tcnative-boringssl-static"
4547
exclude group: "io.netty", module: "netty-tcnative-classes"
48+
exclude group: "org.apache.commons", module: "commons-compress"
4649
exclude group: "org.apache.logging.log4j"
4750
exclude group: "org.fusesource.leveldbjni"
4851
exclude group: "org.lz4"

spark-submission-worker/build.gradle

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,13 @@ dependencies {
2626

2727
implementation(libs.spark.kubernetes) {
2828
exclude group: "com.github.luben"
29+
exclude group: "com.ibm.icu"
30+
exclude group: "commons-codec"
2931
exclude group: "commons-collections", module: "commons-collections"
3032
exclude group: "io.fabric8"
3133
exclude group: "io.netty", module: "netty-tcnative-boringssl-static"
3234
exclude group: "io.netty", module: "netty-tcnative-classes"
35+
exclude group: "org.apache.commons", module: "commons-compress"
3336
exclude group: "org.apache.logging.log4j"
3437
exclude group: "org.fusesource.leveldbjni"
3538
exclude group: "org.lz4"

0 commit comments

Comments
 (0)