Support local testing

[willtemperley]

Describe the enhancement requested

Currently some test files are being generated by CI scripts. This means they are not accessible in a local clone of this repository, making local testing difficult.

The three generated files are very small, less than 1KB each:
testdata_bool.arrow
testdata_double.arrow
testdata_struct.arrow

Can these just be checked into the repo? Or at least document a generation step a developer can run locally.

(As mentioned in #77)

[kou]

Can these just be checked into the repo?

If you want to choose this approach, please rewrite these tests to use Swift not Go instead of this approach.
See also: #78 (comment)

[kou]

Or at least document a generation step a developer can run locally.

FYI: We can build and run tests by docker compose run --rm ubuntu or ci/scripts/build.sh $(pwd) ../arrow-swift.build && ci/scripts/test.sh $(pwd) ../arrow-swift.build locally.

[willtemperley]

If they can be autogenerated in Swift, then I could just generate them in temp files in the test.

[kou]

Let's try the approach.

[willtemperley]

I was looking for a way to write .arrow files. I thought that ArrowWriter().writeFile(info:) [1] would do so, but this does not appear to be writing valid .arrow files.

It appends ARROW1 to the header and footer, however the file is unreadable in any readers I've tried, such as pyarrow.

Looking at testFileWriter_bool in IPCFileReaderTests, it generates a 966 byte file using ArrowWriter, parses it and compare it to the expected results manually in checkBoolRecordBatch . It passes.

The test testFileReader_bool uses the same routine checkBoolRecordBatch to read a 650 byte file generated by Go.

So the flatbuffers message decodes to the same thing but the first bytes in the Go written data are:
0x14000000000000000c0016001

Whereas the padding looks different in the Swift written message:
0x1c0000000000000000000000000000000c0016001

So I think there's almost certainly a writer bug here, and likely an issue with the reader being too lenient.

[1]

arrow-swift/Arrow/Sources/Arrow/ArrowWriter.swift

Line 354 in a1fcb23

public func writeFile(_ info: ArrowWriter.Info) -> Result<Data, ArrowError> {

[kou]

Oh... Can you debug it?

[willtemperley]

Unfortunately I can't really devote any more time to this.

[kou]

No problem. Thanks for helping us.