Skip to content

Commit c8a2643

Browse files
fix: Update RecordExpander to return nothing when path doesn't exist
Changes: - Add back 'else: yield from []' in DpathExtractor for explicit empty case - Update RecordExpander to return nothing when expand_records_from_field path doesn't exist or isn't a list - Update unit tests to expect no records instead of original record when expansion fails This makes RecordExpander stricter: it only emits records when successfully expanding a list. For Stripe invoice_line_items, this ensures we only emit line items, not invoice objects. All 24 tests passing. Requested by @DanyloGL. Co-Authored-By: unknown <>
1 parent b04e174 commit c8a2643

File tree

3 files changed

+6
-6
lines changed

3 files changed

+6
-6
lines changed

airbyte_cdk/sources/declarative/expanders/record_expander.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,11 +58,9 @@ def expand_record(self, record: MutableMapping[Any, Any]) -> Iterable[MutableMap
5858
try:
5959
nested_array = dpath.get(record, expand_path)
6060
except (KeyError, TypeError):
61-
yield record
6261
return
6362

6463
if not isinstance(nested_array, list):
65-
yield record
6664
return
6765

6866
if len(nested_array) == 0:

airbyte_cdk/sources/declarative/extractors/dpath_extractor.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,3 +111,5 @@ def extract_records(self, response: requests.Response) -> Iterable[MutableMappin
111111
yield from self.record_expander.expand_record(extracted)
112112
else:
113113
yield extracted
114+
else:
115+
yield from []

unit_tests/sources/declarative/extractors/test_dpath_extractor.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -191,14 +191,14 @@ def test_dpath_extractor(field_path: List, decoder: Decoder, body, expected_reco
191191
["items"],
192192
False,
193193
{"data": {"id": "parent_1"}},
194-
[{"id": "parent_1"}],
194+
[],
195195
),
196196
(
197197
["data"],
198198
["items"],
199199
False,
200200
{"data": {"id": "parent_1", "items": "not_an_array"}},
201-
[{"id": "parent_1", "items": "not_an_array"}],
201+
[],
202202
),
203203
(
204204
["data"],
@@ -234,8 +234,8 @@ def test_dpath_extractor(field_path: List, decoder: Decoder, body, expected_reco
234234
"test_expand_nested_array",
235235
"test_expand_with_original_record",
236236
"test_expand_empty_array_yields_nothing",
237-
"test_expand_missing_path_yields_original",
238-
"test_expand_non_array_yields_original",
237+
"test_expand_missing_path_yields_nothing",
238+
"test_expand_non_array_yields_nothing",
239239
"test_expand_deeply_nested_path",
240240
"test_expand_mixed_types_in_array",
241241
"test_expand_multiple_parent_records",

0 commit comments

Comments
 (0)