Skip to content

Commit cf43623

Browse files
LuciferYangsrowen
authored andcommitted
[SPARK-36900][SPARK-36464][CORE][TEST] Refactor : size returns correct positive number even with over 2GB data to pass with Java 8, 11 and 17
### What changes were proposed in this pull request? Refactor `SPARK-36464: size returns correct positive number even with over 2GB data` in `ChunkedByteBufferOutputStreamSuite` to reduce the total use of memory for this test case, then this case can pass with Java 8, Java 11 and Java 17 use `-Xmx4g`. ### Why are the changes needed? `SPARK-36464: size returns correct positive number even with over 2GB data` pass with Java 8 but OOM with Java 11 and Java 17. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Pass the Jenkins or GitHub Action - Manual test ``` mvn clean install -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite ``` with Java 8, Java 11 and Java 17, all tests passed. Closes apache#34284 from LuciferYang/SPARK-36900. Authored-by: yangjie01 <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 00b87c9 commit cf43623

File tree

1 file changed

+6
-5
lines changed

1 file changed

+6
-5
lines changed

core/src/test/scala/org/apache/spark/util/io/ChunkedByteBufferOutputStreamSuite.scala

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -121,12 +121,13 @@ class ChunkedByteBufferOutputStreamSuite extends SparkFunSuite {
121121
}
122122

123123
test("SPARK-36464: size returns correct positive number even with over 2GB data") {
124-
val ref = new Array[Byte](1024 * 1024 * 1024)
125-
val o = new ChunkedByteBufferOutputStream(1024 * 1024, ByteBuffer.allocate)
126-
o.write(ref)
127-
o.write(ref)
124+
val data4M = 1024 * 1024 * 4
125+
val writeTimes = 513
126+
val ref = new Array[Byte](data4M)
127+
val o = new ChunkedByteBufferOutputStream(data4M, ByteBuffer.allocate)
128+
(0 until writeTimes).foreach(_ => o.write(ref))
128129
o.close()
129130
assert(o.size > 0L) // make sure it is not overflowing
130-
assert(o.size == ref.length.toLong * 2)
131+
assert(o.size == ref.length.toLong * writeTimes)
131132
}
132133
}

0 commit comments

Comments
 (0)