Writing and Reading Random Access Files

[okartal]

Maybe related to #353

It is already possible to use Tables.partitioner to write record batches to a single Arrow file. However, when I read that file with Arrow.Table I do not know how to access a specific record batch like here: https://arrow.apache.org/docs/java/ipc.html#writing-and-reading-random-access-files

According to the docs, this should be possible but I am not sure if that is not implemented yet or simply not documented.

[quinnj]

You're right that we don't expose this very well (i.e at all) via Arrow.Table right now; but using Arrow.Stream gives you back an iterator of Arrow.Table for each record batch. But we could probably also expose a way via Arrow.Table to let you get the individual tables. Something to think about, or at least improve in the docs mentioning Arrow.Stream.

[okartal]

According to https://arrow.apache.org/docs/python/ipc.html#writing-and-reading-random-access-files we need to use a seek method to implement random access to a batch

[Moelf]

we don't have to do any Python implementation says, that's specifically for Python. A batch is a well defined thing in file format, independent of which implementation we're talking about, it's purely a logical question of how do we get there given the schema / metadata and what's the interface for user

[oschulz]

Is there a way to get the batch-offset table with Arrow.jl, if the data is written in "file" mode?

[Yuan-Ru-Lin]

Is there a way to get the batch-offset table with Arrow.jl, if the data is written in "file" mode?

Yes.

Consider test.arrow generated by the following script.

using Arrow
using TypedTables
using Tables

t = Table(
    a=collect(1:10_000),
    b=rand(Float32, 10_000),
    c=rand(ComplexF32, 10_000),
)

# This would produce 10 RecordBatches
Arrow.write("test.arrow", Tables.partitioner(Iterators.partition(t, 1_000)))

Then one can get the indices of all the RecordBatches by reading the relevant bytes and parsing them using Arrow.FlatBuffers.getrootas(Arrow.Meta.Footer, _footerbytes, 0)

using Arrow

f = open("test.arrow")

# Check whether the magic number is there
seekend(f)
seek(f, position(f) - 6)
@assert String(read(f, 6)) == "ARROW1"

# Fetch footer size
seekend(f)
seek(f, position(f) - 6 - 4)
footersize = only(reinterpret(Int32, read(f, 4)))
@assert footersize == 560

# Fetch footer
seekend(f)
seek(f, position(f) - 6 - 4 - 560)
_footerbytes = read(f, 560)
_footer = Arrow.FlatBuffers.getrootas(Arrow.Meta.Footer, _footerbytes, 0)

"""
julia> _footer.recordBatches
10-element Arrow.FlatBuffers.Array{Arrow.Flatbuf.Block, NTuple{24, UInt8}, Arrow.Flatbuf.Footer}:
 Arrow.Flatbuf.Block(offset = 320, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 20640, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 40960, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 61280, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 81600, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 101920, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 122240, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 142560, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 162880, metaDataLength = 320, bodyLength = 20000)
 Arrow.Flatbuf.Block(offset = 183200, metaDataLength = 320, bodyLength = 20000)
"""

# Sanity check: fetch the first column in the first block using the above information
seek(f, 320 + 320)
block1data = read(f, 20000)
reinterpret(Int64, block1data[1:8000])

"""
julia> reinterpret(Int64, block1data[1:8000])
1000-element reinterpret(Int64, ::Vector{UInt8}):
    1
    2
    3
    4
    (omitted)
"""

I accessed the first batch but in principle one can access to whichever block without reading others.

In order to come up with an API, I still need to know how to parse bytes that make up a RecordBatch.

By the way, this might provide a way to closing #353