Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
825610c
rename flag for executing raw queries to "execute_raw_query"
sh-rp Jun 5, 2025
c149269
return sge queries from the internal _query method which removes a lo…
sh-rp Jun 5, 2025
70116a6
adds some tests to readable dataset and a test for column hint merging
sh-rp Jun 5, 2025
51a91ca
allows any dialect when writing queries and fixes tests
sh-rp Jun 5, 2025
20405f1
update docs and set correct quoting to queries in normalization and l…
sh-rp Jun 5, 2025
c428342
fixes normalizer tests
sh-rp Jun 5, 2025
ec985e4
fix limit on mssql
sh-rp Jun 6, 2025
f153a9b
add missing quote to alias
sh-rp Jun 6, 2025
1238ce0
revert identifier normalization step in normalizer_query and use bigq…
sh-rp Jun 11, 2025
e58ceab
post rebase fix
sh-rp Jun 25, 2025
4af4ef3
smallish pr fixes
sh-rp Jun 25, 2025
1a54a4c
add materializable sqlmodel and handle hints in extractor
sh-rp Jun 25, 2025
7fc4753
add and test always_materialize setting
sh-rp Jun 26, 2025
d8c30a1
add test for sql transformation type
sh-rp Jun 26, 2025
de3d7f0
convert transformation functions to need yield instead of return
sh-rp Jun 26, 2025
30f230d
migrate tests and docs snippets to yield in transformations
sh-rp Jun 26, 2025
f065066
add simple test for materializable model
sh-rp Jun 26, 2025
d33f198
use correct compiler for converting ibis into sqlglot for each dialect
sh-rp Jun 26, 2025
8aa9d7b
add first simple version of using unbound ibis tables in transformations
sh-rp Jun 26, 2025
f6a7b10
skip ibis test on python 3.9
sh-rp Jun 27, 2025
6a77610
Merge branch 'devel' into feat/transformations-more-decoupling
sh-rp Jun 27, 2025
fbd6c2b
fix query building in new relation
sh-rp Jun 27, 2025
1479d5b
return a "real" relation from a transformation
sh-rp Jun 27, 2025
77330c2
add ibis option when getting table from dataset
sh-rp Jun 27, 2025
af42236
update model item format tests to use relation
sh-rp Jun 30, 2025
a6cabe6
* remove one unneeded test (same thing is already tested in transform…
sh-rp Jun 30, 2025
166d446
adds support for adding resource hints to pyarrow items
sh-rp Jun 30, 2025
4c720da
switch most read access tests to default dataset
sh-rp Jun 30, 2025
d4d3285
update datasets and transformations docs pages
sh-rp Jun 30, 2025
f6acbb3
separate ibis and default dbapi datasets and fix typing
sh-rp Jun 30, 2025
de1fdf3
update transformation tests and small typing fixes for updated datasets
sh-rp Jun 30, 2025
5bc6470
fix default dataset type
sh-rp Jun 30, 2025
2881db0
fix wei sqlglot conversion
sh-rp Jun 30, 2025
fb983e3
add sqlglot dialect type and some cleanup
sh-rp Jun 30, 2025
9a5f1b0
fix dataset snippets
sh-rp Jun 30, 2025
4578eb9
fix sqlglot schema test
sh-rp Jun 30, 2025
fde9bd5
removes ibis relation and dataset
sh-rp Jul 1, 2025
34a8eac
fix one doc snippet
sh-rp Jul 1, 2025
9345476
rename dataset and relation interfaces
sh-rp Jul 1, 2025
cea5cb9
fix relation ship between cursor and relation, remove function wiring…
sh-rp Jul 1, 2025
a7b550b
clean up readablerelation (no actual code changes)
sh-rp Jul 1, 2025
9872119
fix str test to assume pretty sql (which it is now)
sh-rp Jul 1, 2025
3d6112e
small changes from review comments:
sh-rp Jul 1, 2025
6bc1f20
rename query method to "to_sql" on relation
sh-rp Jul 1, 2025
d08aa96
clean up transform function a bit (could maybe be even better=
sh-rp Jul 1, 2025
540e395
add support for "non-generator" transformations
sh-rp Jul 1, 2025
7b9e8df
move hints computation into resource class
sh-rp Jul 1, 2025
41e1239
smallish PR fixes
sh-rp Jul 2, 2025
57e09fd
add support for dynamic hints in transformations
sh-rp Jul 2, 2025
0eccfdd
fixes dynamic table caching
rudolfix Jul 2, 2025
a141170
Enhances ReadableDBAPIRelation: min/max, filter with expression (#2833)
anuunchin Jul 4, 2025
15b4a43
Merge branch 'devel' into feat/transformations-more-decoupling
sh-rp Jul 4, 2025
3b46e8a
prevent sqglot schema from adding default hints info, only allow para…
sh-rp Jul 4, 2025
5b906ad
make multi schema transformations work again
sh-rp Jul 4, 2025
44ae438
move model item format tests to transformations folder
sh-rp Jul 4, 2025
089f5ec
re-order interface tests and fix playground dataset access
sh-rp Jul 5, 2025
b06d715
PR review test updated
sh-rp Jul 7, 2025
c5f9b33
update dataset and transformation pages
sh-rp Jul 7, 2025
b6577fd
Merge branch 'devel' into feat/transformations-more-decoupling
sh-rp Jul 9, 2025
63dfa5f
update transformations tests to new fruitshop
sh-rp Jul 9, 2025
83372df
Last PR fixes
sh-rp Jul 11, 2025
bd4cfd7
update columns_schema property
sh-rp Jul 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion dlt/extract/extractors.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,11 @@ def _write_to_dynamic_table(self, resource: DltResource, items: TDataItems, meta
if table_name in self._filtered_tables:
continue
# allow to cache dynamic table hints if only table name is dynamic
if table_name not in self._table_contracts or resource._table_has_other_dynamic_hints:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rudolfix I had to revert his change that you made, because yielding multiple transformations with different schemas would not work without it. Relevant test is: def test_multiple_transformations_in_function. Maybe you should review this once again.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right! but this prevents caching of the computed dynamic table hints which is 99% of cases we handle. my take:
convert _table_has_other_dynamic_hints into has_other_dynamic_hints which for transformation resources is always True and for others it falls back to the current behavior

if (
table_name not in self._table_contracts
or resource.has_dynamic_hints
or resource._table_has_other_dynamic_hints
):
item = self._compute_and_update_tables(
resource, table_name, item, TableNameMeta(table_name)
)
Expand Down
Loading