Skip to content

Commit 41f10f1

Browse files
fix(concurrent): use last slice for client-side incremental filtering
The _get_concurrent_state() method was using slices[0] (the earliest slice) instead of slices[-1] (the most recent slice) when extracting the cursor value from partitioned state. Since merge_intervals() sorts slices in ascending order by (START_KEY, END_KEY), using the first slice caused should_be_synced() to use the earliest timestamp as the lower bound, admitting ALL records instead of only new records. This bug affected all low-code connectors using DatetimeBasedCursor with is_client_side_incremental: true, including Notion's pages, databases, comments, and blocks streams. Fixes: airbytehq/oncall#8854 Co-Authored-By: unknown <>
1 parent daf7d48 commit 41f10f1

File tree

1 file changed

+7
-5
lines changed
  • airbyte_cdk/sources/streams/concurrent

1 file changed

+7
-5
lines changed

airbyte_cdk/sources/streams/concurrent/cursor.py

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -233,12 +233,14 @@ def _get_concurrent_state(
233233

234234
value_from_partitioned_state = None
235235
if slices_from_partitioned_state:
236-
# We assume here that the slices have been already merged
237-
first_slice = slices_from_partitioned_state[0]
236+
# We assume here that the slices have been already merged.
237+
# After merging, slices are sorted in ascending order by (START_KEY, END_KEY),
238+
# so the last slice contains the most recent cursor value for client-side filtering.
239+
last_slice = slices_from_partitioned_state[-1]
238240
value_from_partitioned_state = (
239-
first_slice[self._connector_state_converter.MOST_RECENT_RECORD_KEY]
240-
if self._connector_state_converter.MOST_RECENT_RECORD_KEY in first_slice
241-
else first_slice[self._connector_state_converter.END_KEY]
241+
last_slice[self._connector_state_converter.MOST_RECENT_RECORD_KEY]
242+
if self._connector_state_converter.MOST_RECENT_RECORD_KEY in last_slice
243+
else last_slice[self._connector_state_converter.END_KEY]
242244
)
243245
return (
244246
value_from_partitioned_state

0 commit comments

Comments
 (0)