[SPARK-53637][BUILD] Demote bcprov-jdk18on to test scope

### What changes were proposed in this pull request?

This PR logically reverts SPARK-51311.

### Why are the changes needed?

HADOOP-19152 has already been included in Hadoop 3.4.2 via apache/hadoop#7439, the previous workaround is unnecessary anymore.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

I manually tested it in a kerberized cluster with a SparkPi.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #52390 from pan3793/SPARK-53637.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: pan3793

LICENSE-binary

@@ -479,7 +479,6 @@ dev.ludovic.netlib:blas
 dev.ludovic.netlib:arpack
 dev.ludovic.netlib:lapack
 net.razorvine:pickle
-org.bouncycastle:bcprov-jdk18on
 org.checkerframework:checker-qual
 org.typelevel:algebra_2.13:jar
 org.typelevel:cats-kernel_2.13

assembly/pom.xml

@@ -136,16 +136,6 @@
       <artifactId>guava</artifactId>
       <scope>${hadoop.deps.scope}</scope>
     </dependency>
-
-    <!--
-      SPARK-51311: HDFS-15098 (3.4.0) adds hard dependency on bcprov-jdk18on, Spark fails to submit
-      to Kerberized cluster without this dependency, until HADOOP-19152 (3.5.0, unreleased)
-      -->
-    <dependency>
-      <groupId>org.bouncycastle</groupId>
-      <artifactId>bcprov-jdk18on</artifactId>
-      <scope>${hadoop.deps.scope}</scope>
-    </dependency>
   </dependencies>
 
   <build>

dev/deps/spark-deps-hadoop-3-hive-2.3

@@ -27,7 +27,6 @@ avro/1.12.0//avro-1.12.0.jar
 azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
 azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar
 azure-storage/7.0.1//azure-storage-7.0.1.jar
-bcprov-jdk18on/1.81//bcprov-jdk18on-1.81.jar
 blas/3.0.4//blas-3.0.4.jar
 breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
 breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar