Replies: 1 comment
-
Yes, lazy chunks are perfect for that. Look at the format of the lazychunk to understand how bstarts and bsizes are stored. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I want to do partial reads of a remote file compressed with blosc2, transferring as few bytes as possible. Given a list of (offset,extent) ranges in the decompressed output, I'd like to be able to generate a list of (offset, extent) ranges in the compressed file to fetch and then somehow decompress this data with blosc2. I'd like to do this in as few round-trips as possible, ideally one to fetch the header + bstarts, and one to fetch the ranges of compressed data. I have complete control over the generation of the compressed file, and it's also an option to pre-process them somehow (e.g. extracting the header + bstarts out into a separate file for easy lookup).
Is this at all possible with the public api? Is there some example code I should look at? So far I'm thinking about calling
blosc2_schunk_get_chunk
for each chunk during preprocessing, saving that somewhere, and then usingblosc2_io_cb
to define my own reader to read from the downloaded fragments.Beta Was this translation helpful? Give feedback.
All reactions