[python] introduce update_columns in python api. #6861
Conversation
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Please document this too.
|
@zhoulii Can you find a rough reference for other systems? This API definition provides a global RowID. |
|
Maybe new API: |
rust/lance/src/dataset/fragment.rs:1605 lance provides update_columns api that can accept data with rowids, but it's more complicated, it can update columns based on other conditions. |
Thanks for the review ! @JingsongLi Comments are addressed, please take a look again, thanks. |
| self._partial_column_write = PartialColumnWrite(self.table, self.commit_user) | ||
|
|
||
| def write_arrow(self, table: pa.Table): | ||
| if self._partial_column_write is not None: |
There was a problem hiding this comment.
Can you abstract file_store_write to avoid so many if else?
There was a problem hiding this comment.
I haven't figured out the exact approach yet, so I just deleted the else branch for now.
| file_store_write.write(partition_tuple, 0, batch) | ||
|
|
||
| # Prepare commit and assign first_row_id | ||
| commit_messages = file_store_write.prepare_commit(BATCH_COMMIT_IDENTIFIER) |
There was a problem hiding this comment.
We should set sequence number to define the version.
There was a problem hiding this comment.
sequence number will be set in commit stage.
| """Find the partition for a given first_row_id using pre-built partition map.""" | ||
| return self.first_row_id_to_partition_map.get(first_row_id) | ||
|
|
||
| def _write_group(self, partition: GenericRow, first_row_id: int, |
There was a problem hiding this comment.
What should I do if all the data in the file is not included?
There was a problem hiding this comment.
Added some row counts validations, if not match, an error would be raised.
zhoulii
force-pushed
the
sort-partial-update
branch
from
99bacc9 to
6ae9bff
Compare
2 participants
Purpose
Linked issue: close #6859
Tests
API and Format
Documentation