python way to convert parquet into vortex which does not load whole parquet into memory

[antonkulaga]

From what I saw in python integration docs to convert parquet into vortex I need to load everything in memory. But what if parquet files are larger than RAM I have, could you suggest the code which will do it for me?

[a10y]

The writing function supports streaming from a pyarrow RecordBatchReader: https://docs.vortex.dev/api/python/io#vortex.io.write

If you're just looking to do simple conversion, you can also use the vx CLI binary for this:

cargo install vortex-tui

vx convert file.parquet

will create a new file file.vortex with all of the data, in streaming fashion.

[a10y]

See also this prior issue: #4160