Add a multifile scan operator

[joseph-isaacs]

Should we add a MultiFile scan operator to vortex, or should be this functionality in the duckdb crate.

It would contain the following features:

• Read whole directories using glob syntax.
• Unify file schemas by index or column name.
• Support file_index query exprs.

This operator is responsible for orchestrating the concurrent read of many files.

Ideally this will support external many thread models (e.g. duckdb using n-threads to drive the scan).

[gatesn]

I believe there's already an issue for this. And yeah, should live alongside the scan. We actually already have a vortex-scan crate from the past if you wanted to pull out the existing single file scan also

[onursatici]

is supporting external threading models an orthogonal concern? We can also apply that to the current scan it feels like

[0ax1]

Yes, that needs to be considered as part of this. E.g. DuckDB provides the threads the operator can utilize.

[gatesn]

is supporting external threading models an orthogonal concern? We can also apply that to the current scan it feels like

Yes it is somewhat orthogonal, but a multi-file scan should ideally make progress on one file per-core. So it's worth thinking about how we can support that model.

Whereas a multi-file layout reader would have a hard time spawning its work from different files onto known-different threads. Not sure if I'm explaining that well...

[onursatici]

yea agreed, also agree on this being a scan level thing rather than a layout reader level thing. I guess what I mean is solving the problem of core affinity per file in a multi file scan would require us to solve core affinity per column in a single file scan, and that is a general scan level problem rather than a multi file scan only problem

[danking]

I gave some thought to Spiral using this and I think the two use-cases are too different. Spiral (and I guess Iceberg) will want to read file-level statistics from the manifest file. For large manifests you might even want to push stats filters into the read of the manifest files.

Spiral also uses a lot of special logic during the read which I think might be hard to glue onto this.