link to the sigmod paper

[photoszzt]

I want to read about the sigmod paper. Is there a link to it? I can't find it on the SIGMOD website and ACM DL.

[sighingnow]

The full paper will be available at the ACM DL shortly: https://dl.acm.org/doi/proceedings/10.1145/3555041

[photoszzt]

Ok, I'm wondering what kind of macro benchmark do you guys run? The benchmark dir only contains the malloc benchmark. The readme talks about the need for sharing graph data structure and tensor. Is there a repo that shows this use case? Is the artifact available?

[photoszzt]

Looking the interface and guarantee, this project has many similarities with Ray. What's the main differences between Ray and v6d?

[sighingnow]

For how distributed tensors/dataframes are shared with vineyard, you could refer to https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/dask/dask.py for the basic idea. For how graphs are shared in vineyard, see https://github.com/v6d-io/v6d/blob/main/modules/graph/fragment/arrow_fragment.vineyard-mod which is an implementation for property graphs and has been used in graphscope (https://github.com/alibaba/GraphScope) for performing different kinds of computation (traversal, analytical, GNN) on the same piece of memory.

The benchmark itself is currently mixed with micro benchmarks and end-to-end benchmarks where some internal datasets, tables and scripts are used. There indeed an (old) issue that tracks part of the benchmark suite (#721). The artifacts for micro-benchmarks will be available after the meeting (being busy days for preparing the presentation).

[sighingnow]

Looking the interface and guarantee, this project has many similarities with Ray. What's the main differences between Ray and v6d?

Unlike ray targets a broad landscape, vineyard itself targets the data sharing part in data analytical workflows. You may wonder it looks quite confusing with the ray object-store.

Vineyard doesn't require integrating the execution workflow/application to vineyard, you can imagine vineyard is a kind of key-value store like Redis where the value can be complex (even distributed) objects where the application is not tightly coupled with vineyard can be MPI programs, single-process jobs, dataflow models, or even any customized distributed jobs. You know, its non-trivial to move an existing complex application entirely to ray, but it is relatively easy to insert some kinds of put/get instructions to the existing codebase.

There's another key difference with key is how to model and store objects. Ray requires serializing the object to bytes and stores using plasma. It's not straightforward (and inefficient) to serialize many complex but commonly used objects to bytes, for example, flat hashmap and graphs. Another key differents is for large distributed objects, it is impossible to serialize to fit the main memory of a single machine. Vineyard doesn't have these limitations, as it models objects as metadata plus a set of blobs.

We have noticed ray recently has added abstractions like tables and datasets to ease the data manipulation of distributed objects. Under the hood it still needs serialization.