How does Vortex handle storing large blobs and vector embeddings?

[rahil-c]

Hi all, I primarily work in the big data java ecosystem and am new to Vortex.

I was wondering what is the recommendation that the Vortex team gives users when it comes to storing large binary content such as (images, videos) as well the vector embeddings within vortex files.

When checking the Vortex type system https://docs.vortex.dev/concepts/dtypes#logical-types, I see there is a BINARY type for holding blobs as well a LIST type that can be used likely for an array of float for holding the embedding. Is there any special flags or features when it comes to how blobs or vectors are managed by vortex?

When looking at the spark integration saw these mappings https://github.com/vortex-data/vortex/blob/develop/java/vortex-spark/src/main/java/dev/vortex/spark/SparkTypes.java, which seems to align with what I mentioned but wanted to know if there is any other insights on how users can best leverage vortex for storing this kinda of data?

[danking]

Hey @rahil-c ! Welcome to the orange side of the big data ecosystem 😉

Indeed Vortex supports bytes arrays, list arrays, and also fixed-length-list arrays. You probably want the latter for vectors. Looks like we need to add support for fixed-length-list to the Java bindings.

Below, I share some first-hand information about a dataset with vectors and matrices, but, to best answer your question, I think I need to know what kinds of operations you want to perform on this table of vectors / images / videos. Vortex expressions are extensible, but we do not currently have anything like "slice video" or "slice image", which, I imagine, are important to you. I suspect storing a PNG as binary is not a very useful thing to do (because you're treating very meaningful bytes as meaningless!).

PS: if you do choose to store raw binary in Vortex, you probably want to enable the compact compressor which enables paged ZSTD compression for binary columns.

---

FWIW, we have a benchmark which stores a matrix of human genomes in a Vortex array. Each row of the array contains a few thousand metadata fields (about that genetic position) as well as nine vector or matrix fields (which have an entry or column per sample of which there are ~4k). The types of these fields look like this (from cargo run --bin vx tree layout):

├── GT: vortex.stats, dtype: list(u64?)?, children: 2
│   ├── data: vortex.chunked, dtype: list(u64?)?, children: 2
│   │   ├── [0]: vortex.flat, dtype: list(u64?)?
│   │   └── [1]: vortex.flat, dtype: list(u64?)?

Vortex, by default, does not use row groups. These vector and matrix columns, which are significantly larger than the metadata, all end up at the beginning of the file (from cargo run --bin vx inspect). The [0] indicates that this array is stored in multiple chunks and this is the first chunk.

Segment details:

   #      Start..End          Length  Align  Path
   0          8..9177412     9177404      8  GT.data.[0]
   1    9177416..32362252   23184836      8  GQ.data.[0]
   2   32362256..63244228   30881972      8  DP.data.[0]
   3   63244232..73845564   10601332      8  AD.data.[0]
   4   73845568..104617620  30772052      8  MIN_DP.data.[0]
   5  104617624..114226084   9608460      8  PL.data.[0]
   6  114226088..119240700   5014612      8  SB.data.[0]

The metadata arrays are stored near the end of the file without chunking due to their extremely small size:

2308  168593248..168647948     54700      8  AS_FS.data
2309  168647952..168648196       244      8  AS_FS.zones
2310  168648200..168688516     40316      8  AS_MQ.data
2311  168688520..168688676       156      8  AS_MQ.zones
2312  168688680..168731060     42380      8  AS_MQRankSum.data
2313  168731064..168731308       244      8  AS_MQRankSum.zones
2314  168731312..168779532     48220      8  AS_pab_max.data
2315  168779536..168779780       244      8  AS_pab_max.zones
2316  168779784..168820132     40348      8  AS_QUALapprox.data
2317  168820136..168820292       156      8  AS_QUALapprox.zones
2318  168820296..168860612     40316      8  AS_QD.data
2319  168860616..168860772       156      8  AS_QD.zones

If you run cargo run --bin vx browse and hit TAB you'll get an interactive visualization of the layout (though its organized by column rather than by position within the file).

[rahil-c]

Thank you @danking for an indepth response ill need to read more to better understand more on vortex to better understand this but appreciate the response.

[robert3005]

We are still missing a list layout that would properly partition large strings and lists if they don’t end up dictionary encoded. There’s also some harder to implement features like storing big values outside of the file which we need for values >4GB

[rahil-c]

Thank you @robert3005 for letting me know. For now then are users directly storing pointers to locations of large blobs in cloud storage?

[robert3005]

Yes, right now it's on the users of vortex

[infogulch]

Ok but like what's the scope of this feature request? To me, "storing big blobs outside the file" in a way that it's not "on the users of vortex" means relative paths, unc paths, urls, network requests, range requests, authentication x10, platform specific cloud storage apis x4, network failure modes, timeouts, then all of this needs to be exposed in the vortex application api... I dunno man, it seems like this feature request drops a nuke on the project scope, basically "just reimplement all of DataFusion".

But maybe I'm just being dense, do you have a more reasonable interpretation of the scope of this feature? (I hope it's not "just implement all that but only for the set of technologies that I use: a, b, c, d")

[gatesn]

There's two versions of this:

1. Storing "medium" blobs. ~1kb up to ~10s MB that one might reasonably expect to store inline in a columnar file format. This includes small images, audio, feature vectors / tensors, etc.
2. Storing "large" blobs. ~100s MB up to GBs. Honestly, I would deem this out-of-scope for Vortex files to store these things inline. That said, I do know of at least one example of a custom Vortex expression that can download URLs. So extending Vortex with the functionality to run a scan using a projection expression of {bytes: s3.fetch($.url)} where the s3.fetch expression downloads the files from $.url column is entirely possible.

It seems like this discussion originally started to tackle (1), so I think we should stick to that interpretation. In that case, Vortex currently has poor support for partitioning columns with large variable-length data. The default write strategy will choose chunks of 8k rows, which can often be very large when each row stores a 1kb vector.