Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit 718bbc9

Browse files
BryanCutlerueshin
authored andcommitted
[SPARK-22067][SQL] ArrowWriter should use position when setting UTF8String ByteBuffer
## What changes were proposed in this pull request? The ArrowWriter StringWriter was setting Arrow data using a position of 0 instead of the actual position in the ByteBuffer. This was currently working because of a bug ARROW-1443, and has been fixed as of Arrow 0.7.0. Testing with this version revealed the error in ArrowConvertersSuite test string conversion. ## How was this patch tested? Existing tests, manually verified working with Arrow 0.7.0 Author: Bryan Cutler <[email protected]> Closes apache#19284 from BryanCutler/arrow-ArrowWriter-StringWriter-position-SPARK-22067.
1 parent ee13f3e commit 718bbc9

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,8 +234,9 @@ private[arrow] class StringWriter(val valueVector: NullableVarCharVector) extend
234234

235235
override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
236236
val utf8 = input.getUTF8String(ordinal)
237+
val utf8ByteBuffer = utf8.getByteBuffer
237238
// todo: for off-heap UTF8String, how to pass in to arrow without copy?
238-
valueMutator.setSafe(count, utf8.getByteBuffer, 0, utf8.numBytes())
239+
valueMutator.setSafe(count, utf8ByteBuffer, utf8ByteBuffer.position(), utf8.numBytes())
239240
}
240241
}
241242

0 commit comments

Comments
 (0)