Comparison to VirtualiZarr

[TomNicholas]

Hi, I develop VirtualiZarr, a package that seems to have very similar goals to dataplug. I found out about this library via your SciPy talk announcement - I also have a SciPy talk about VirtualiZarr!

VirtualiZarr was developed to replace the Kerchunk package, to serve as a way to make non-cloud-optimized file formats accessible in a cloud-optimized manner through the Zarr API. It's based on the idea that most binary chunked array file formats can be mapped to the Zarr data model. Both Kerchunk and Zarr are mentioned in your talk abstract.

Our packages seem very similar. As far I can tell, they both:

1. Target non-cloud-optimized data sat in object storage
2. Pre-process that data to extract metadata and chunk / partition references
3. Can perform that pre-processing at scale in parallel (VirtualiZarr can use dask or lithops or a general parallel executor, though this functionality hasn't been released yet)
4. Persist those chunk / partition references to storage in new objects
5. Allow cloud-optimized parallel access to the original data by having the data user hit the serialized chunk / partition references first.

Some possible differences:

• VirtualiZarr is targeting array data, and maps all data to the Zarr data model (of a heirarchy of multidimensional chunked arrays). It seems dataplug is more general but produces less structured output (like Kerchunk could do in theory)?
• VirtualiZarr can assemble references from a large number of files into one big cloud-optimized "virtual datacube". It's unclear to me if dataplug tries to do that.
• Dataplug puts a lot of emphasis on re-partitioning. If I understand correctly this is analogous to this idea in VirtualiZarr of slicing into uncompressed chunks (which I haven't implemented yet).
• VirtualiZarr can write to multiple formats, specifically Kerchunk and Icechunk.
• VirtualiZarr works with Icechunk, which can be thought of as serverless transactional database of chunk references.
• Each library currently understands how to parse some file formats that the other doesn't. See Listing every format that could be represented as virtual zarr zarr-developers/VirtualiZarr#218.

I'm curious if my assessment is correct, and if so whether there is any opportunity to join forces 😀

[abourramouss]

Hi @TomNicholas, thanks for opening the Issue!

I am not part of URV anymore, which is the university in where dataplug was developed, but I can give you an answer.

As you said, virtualizarr is more focused on array data, multidimensional data cubes. Dataplug started as a tool to partition unstructured formats (first formats were fastq,fasta and csv), and currently supports a bunch more. The idea is to generate a reference of data in parallel (or whenever it's not possible, sequentially) . You could repartition into any number of arbitrary chunks without the need to preprocess the data again.
The output after preprocessing is a unstructured file as pointed out, and it gives enough information to do "atomic" partitions, i.e if it's a csv format, to not split in between lines.

One big advantage that virtualizarr has over dataplug is the zarr model, correct me if I am wrong, but data ingested with virtualizarr is converted to zarr (Unless zarr now supports byte-range references)? and you get the advantage of having a uniform api for processing? So answering your question, dataplug produces a less structured output and does not provide a "compute" api.

dataplug writes into same format the original format was. At some point I tried to make dataplug write zarr outputs from csv, as an idea to have a uniform output and set of operations, but gave up.

I think we talked about dataplug 2 years ago in a cubed issue (cubed-dev/cubed#268).

[TomNicholas]

unstructured formats

You could repartition into any number of arbitrary chunks without the need to preprocess the data again.

By unstructured you must mean "does not have compressed chunks"? Because you can't index into a subset of a chunk without decompressing it...

data ingested with virtualizarr is converted to zarr (Unless zarr now supports byte-range references)?

The latter. VirtualiZarr examines the given file format and works out what the byte-range references for zarr-compatible access are.

and you get the advantage of having a uniform api for processing?

Yes, and at data read type you're reading all the byte ranges using zarr-python, so it's not just the API that's uniform, it's the implementation too.

You might also be interested in https://github.com/abdenlab/oxbow.

I think we talked about dataplug 2 years ago in a cubed issue (cubed-dev/cubed#268).

Oh huh! I never saw that issue.

[abourramouss]

By unstructured you must mean "does not have compressed chunks"? Because you can't index into a subset of a chunk without decompressing it...

Sorry for the ambiguous language, by unstructured what I meant is data that does not fit into a database or follows a schema.

What I meant was that dataplug was conceived for this type of data (unstructured).

Because you can't index into a subset of a chunk without decompressing it...

Yes true, I think we had some gzip preprocessor that was a bit more efficient, but I don't remember the specifics, you might want to check it out in the preprocessing folder.

The latter. VirtualiZarr examines the given file format and works out what the byte-range references for zarr-compatible access are.

It's a recent feature, isn't it? I remember that zarr references were not a thing, are the references written on the zarr file itself?

Yes, and at data read type you're reading all the byte ranges using zarr-python, so it's not just the API that's uniform, it's the implementation too.

That's one great advantage. I know tiledb did something similar with their format, dataplug does not have a uniform API for compute (as in the set of operations you can do over data), and you still have to do a lot of manual I/o or depend on external tools libraries to read the format and query it on the worker, so the user has to think at 2 levels, distributed and in the node.

Also what do you mean by implementation? That all multidimensional data cube formats implement the same set of operations?

You might also be interested in https://github.com/abdenlab/oxbow.

Thank you, will have a look.

[TomNicholas]

It's a recent feature, isn't it? I remember that zarr references were not a thing, are the references written on the zarr file itself?

Yes. Confusingly it is also not a feature of the "native" Zarr format itself, if you serialize the references to a zarr-compatible format such as Icechunk. VirtualiZarr's ManifestStore class holds the references in-memory, and can write them to Icechunk. It's all "Zarr", but not simply the conventional zarr format + zarr-python, other components are needed.

Also what do you mean by implementation? That all multidimensional data cube formats implement the same set of operations?

I think I misspoke here. I meant that you can use zarr-python's API to access the data. The underlying code that actually issues the GET requests to object storage might need to be different in order to understand the references.

[kikemolina3]

Hi @TomNicholas ,

Let me introduce myself. I'm Enrique Molina, and I'll be the Dataplug (and Lithops) speaker next week at the Scipy Conference.

I wish I'd been up on this issue, but I'm mostly focused on other projects, and it slipped my mind. Our former member @abourramouss did a great job, thanks! Despite having other lines of work, I was assigned a few weeks ago to present this technology at Scipy, so we'll see you soon.

I like seeing how (albeit with different approaches: Dataplug primarily for unstructured data; VirtualiZarr for Zarr datacubes) we're both pursuing the idea of ​​enabling cloud-optimized features in formats that initially aren't.

Let's take advantage of the conference to discuss synergies/possibilities of joining forces, and more. At the same time, I'm going to invest some time in getting to know VirtualiZarr: we'll have an interesting discussion, to say the least.

You can contact me directly via email: enrique.molina@urv.cat

See you soon,

[TomNicholas]

Thanks! I just emailed you.