RecordBatchReader.from_batches currently requires a Sequence (maps to Vec<PyRecordBatch> in PyO3), but the underlying RecordBatchIterator::new only needs an iterator. This makes it impossible to stream lazily from a Python generator into write_parquet without materializing all batches in memory first.
Would it be possible to add a constructor that accepts a Python iterator/generator? Something like:
def gen():
for chunk in chunks:
yield RecordBatch(...)
reader = RecordBatchReader.from_batches(schema, gen())
write_parquet(reader, path, ...)
Happy to submit a PR if you're open to it.