feat(arrow-avro): HeaderInfo to expose OCF header#9548
Open
mzabaluev wants to merge 3 commits intoapache:mainfrom
Open
feat(arrow-avro): HeaderInfo to expose OCF header#9548mzabaluev wants to merge 3 commits intoapache:mainfrom
HeaderInfo to expose OCF header#9548mzabaluev wants to merge 3 commits intoapache:mainfrom
Conversation
Add HeaderInfo to expose OCF header information such as the writer schema and sync marker. Add read_header_info function to the reader module, and its async counterpart to the reader::async_reader module, to read the header from the file reader and return HeaderInfo. Add build_with_header method to async reader builder to enable reuse of the header with multiple readers.
github-actions
bot
added
arrow
mzabaluev
changed the title
HeaderInfo to expose OCF header
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Rework of #9462 along the lines proposed in #9462 (comment).
What changes are included in this PR?
Add
HeaderInfoas a cheaply cloneable value to expose header information parsed from an Avro OCF file.Add
read_header_infofunction to thereadermodule, and its async counterpart to thereader::async_readermodule, to read the header from the file reader and returnHeaderInfo.Add
build_with_headermethod to async reader builder to enable reuse of the header with multiple readers.Are these changes tested?
Added a test for the async reader.
Are there any user-facing changes?
New API in arrow-avro:
reader::HeaderInforeader::read_header_infoandreader::async_reader::read_header_infobuild_with_headermethod ofAvroAsyncFileReader's builder.