|
17 | 17 | under the License. |
18 | 18 | --> |
19 | 19 |
|
20 | | -# RaySQL Design Documentation |
| 20 | +# DataFusion Ray Design Documentation |
21 | 21 |
|
22 | | -RaySQL is a distributed SQL query engine that is powered by DataFusion. |
| 22 | +DataFusion Ray is a distributed SQL query engine that is powered by DataFusion and Ray. |
23 | 23 |
|
24 | 24 | DataFusion provides a high-performance query engine that is already partition-aware, with partitions being executed |
25 | | -in parallel in separate threads. RaySQL provides a distributed query planner that translates a DataFusion physical |
| 25 | +in parallel in separate threads. DataFusion Ray provides a distributed query planner that translates a DataFusion physical |
26 | 26 | plan into a distributed plan. |
27 | 27 |
|
28 | 28 | Let's walk through an example to see how that works. We'll use [SQLBench-H](https://github.com/sql-benchmarks/sqlbench-h) |
@@ -83,9 +83,6 @@ DataFusion's physical plan lists all the files to be queried, and they are organ |
83 | 83 | parallel execution within a single process. In this example, the level of concurrency was configured to be four, so |
84 | 84 | we see `partitions={4 groups: [[ ... ]]` in the leaf `ParquetExec` nodes, with the filenames listed in four groups. |
85 | 85 |
|
86 | | -_DataFusion will soon support parallel execution for single Parquet files but for now the parallelism is based on |
87 | | -splitting the available files into separate groups, so RaySQL will not yet scale well for single-file inputs._ |
88 | | - |
89 | 86 | Here is the full physical plan for query 3. |
90 | 87 |
|
91 | 88 | ```text |
@@ -123,7 +120,7 @@ GlobalLimitExec: skip=0, fetch=10 |
123 | 120 | ## Partitioning & Distribution |
124 | 121 |
|
125 | 122 | The partitioning scheme changes throughout the plan and this is the most important concept to |
126 | | -understand in order to understand RaySQL's design. Changes in partitioning are implemented by the `RepartitionExec` |
| 123 | +understand in order to understand DataFusion Ray's design. Changes in partitioning are implemented by the `RepartitionExec` |
127 | 124 | operator in DataFusion and are happen in the following scenarios. |
128 | 125 |
|
129 | 126 | ### Joins |
@@ -155,7 +152,7 @@ Sort also has multiple approaches. |
155 | 152 | - The input partitions can be collapsed down to a single partition and then sorted |
156 | 153 | - Partitions can be sorted in parallel and then merged using a sort-preserving merge |
157 | 154 |
|
158 | | -DataFusion and RaySQL currently the first approach, but there is a DataFusion PR open for implementing the second. |
| 155 | +DataFusion and DataFusion Ray currently choose the first approach, but there is a DataFusion PR open for implementing the second. |
159 | 156 |
|
160 | 157 | ### Limit |
161 | 158 |
|
@@ -260,13 +257,12 @@ child plans, building up a DAG of futures. |
260 | 257 |
|
261 | 258 | ## Distributed Shuffle |
262 | 259 |
|
263 | | -The output of each query stage needs to be persisted somewhere so that the next query stage can read it. Currently, |
264 | | -RaySQL is just writing the output to disk in Arrow IPC format, and this means that RaySQL is not truly distributed |
265 | | -yet because it requires a shared file system. It would be better to use the Ray object store instead, as |
266 | | -proposed [here](https://github.com/datafusion-contrib/ray-sql/issues/22). |
| 260 | +The output of each query stage needs to be persisted somewhere so that the next query stage can read it. |
| 261 | + |
| 262 | +DataFusion Ray uses the Ray object store as a shared file system, which was proposed [here](https://github.com/datafusion-contrib/ray-sql/issues/22) and implemented [here](https://github.com/datafusion-contrib/ray-sql/pull/33). |
267 | 263 |
|
268 | 264 | DataFusion's `RepartitionExec` uses threads and channels within a single process and is not suitable for a |
269 | | -distributed query engine, so RaySQL rewrites the physical plan and replaces the `RepartionExec` with a pair of |
| 265 | +distributed query engine, so DataFusion Ray rewrites the physical plan and replaces the `RepartionExec` with a pair of |
270 | 266 | operators to perform a "shuffle". These are the `ShuffleWriterExec` and `ShuffleReaderExec`. |
271 | 267 |
|
272 | 268 | ### Shuffle Writes |
|
0 commit comments