-
Notifications
You must be signed in to change notification settings - Fork 4k
Open
Description
I converted from using pyarrow.dataset.write_dataset to pyarrow.parquet.write_to_dataset and noticed that this section of code is overwriting my max_row_group_size param to None. It seems it forces the user to pass in chunksize or row_group_size. I think an additional kwargs.pop would be necessary here.
arrow/python/pyarrow/parquet/core.py
Lines 2194 to 2196 in 71a7b55
| write_dataset_kwargs['max_rows_per_group'] = kwargs.pop( | |
| 'row_group_size', kwargs.pop("chunk_size", None) | |
| ) |
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels