transformations - updates (#2718)

* rename flag for executing raw queries to "execute_raw_query"

* return sge queries from the internal _query method which removes a lot of unneeded transpiling
clean up make_transformation function
tests still pending

* adds some tests to readable dataset and a test for column hint merging

* allows any dialect when writing queries and fixes tests

* update docs and set correct quoting to queries in normalization and load stage

* fixes normalizer tests

* fix limit on mssql
normalize aliases in normalization step

* add missing quote to alias

* revert identifier normalization step in normalizer_query and use bigquery compiler for bigquery destinations

* post rebase fix

* smallish pr fixes

* add materializable sqlmodel and handle hints in extractor

* add and test always_materialize setting

* add test for sql transformation type

* convert transformation functions to need yield instead of return

* migrate tests and docs snippets to yield in transformations

* add simple test for materializable model

* use correct compiler for converting ibis into sqlglot for each dialect
fixes on transformation test

* add first simple version of using unbound ibis tables in transformations

* skip ibis test on python 3.9

* fix query building in new relation

* return a "real" relation from a transformation

* add ibis option when getting table from dataset
natively support unbound ibis tables in transformations and when getting relations from dataset

* update model item format tests to use relation

* * remove one unneeded test (same thing is already tested in transformations)
* fix wei conversion in linneage

* adds support for adding resource hints to pyarrow items

* switch most read access tests to default dataset

* update datasets and transformations docs pages

* separate ibis and default dbapi datasets and fix typing

* update transformation tests and small typing fixes for updated datasets

* fix default dataset type

* fix wei sqlglot conversion

* add sqlglot dialect type and some cleanup

* fix dataset snippets

* fix sqlglot schema test

* removes ibis relation and dataset
consolidates relation and dataset baseclasses with implementations
updates interfaces/protocols fro relation and dataset and makes those the publicly available interface with "Relation" and "Dataset"
remove query method from relation interface

* fix one doc snippet

* rename dataset and relation interfaces

* fix relation ship between cursor and relation, remove function wiring hack in favor of explicit forwarding for better typing

* clean up readablerelation (no actual code changes)

* fix str test to assume pretty sql (which it is now)
fix one transformation snippet

* small changes from review comments:
* query method on dataset
* typing update of table method

* rename query method to "to_sql" on relation

* clean up transform function a bit (could maybe be even better=
reject non-sql strings in transformation to not shadow errors

* add support for "non-generator" transformations

* move hints computation into resource class

* smallish PR fixes

* add support for dynamic hints in transformations
-> this allows to have multiple relations with different schemas in the relation, so this is allowed now too

* fixes dynamic table caching

* Enhances ReadableDBAPIRelation: min/max, filter with expression (#2833)

* Min max, filter with expr_or_string

* Fix in min max test

* Overload fix and docs

* Test read interfaces partially uses default relation max

* prevent sqglot schema from adding default hints info, only allow parametrized types and don't supply hints if none are present in dlt schema

* make multi schema transformations work again

* move model item format tests to transformations folder

* re-order interface tests and fix playground dataset access

* PR review test updated

* update dataset and transformation pages

* update transformations tests to new fruitshop

* Last PR fixes

* update columns_schema property

---------

Co-authored-by: Marcin Rudolf <rudolfix@rudolfix.org>
Co-authored-by: anuunchin <88698977+anuunchin@users.noreply.github.com>

Author: 3 people

.github/workflows/test_common.yml

@@ -110,7 +110,7 @@ jobs:
           pytest tests/pipeline/test_pipeline_extra.py -k arrow ${{ matrix.pytest_args }}
 
       - name: Install pipeline and sources dependencies
-        run: uv sync ${{ matrix.uv_sync_args }} --extra duckdb --extra cli --extra parquet --extra deltalake --extra sql_database --group sentry-sdk --group pipeline --group sources
+        run: uv sync ${{ matrix.uv_sync_args }} --extra duckdb --extra cli --extra parquet --extra deltalake --extra sql_database --group sentry-sdk --group pipeline --group sources --group ibis
 
       - name: Run extract and pipeline tests
         run: |

dlt/__init__.py

@@ -30,7 +30,8 @@
 from dlt.extract.decorators import source, resource, transformer, defer
 from dlt.destinations.decorators import destination
 from dlt.transformations.decorators import transformation
-from dlt.destinations.dataset import dataset, ReadableDBAPIDataset as Dataset
+from dlt.common.destination.dataset import Dataset, Relation
+from dlt.destinations.dataset import dataset
 
 from dlt.pipeline import (
     pipeline as _pipeline,
@@ -81,6 +82,7 @@
     "sources",
     "destinations",
     "Dataset",
+    "Relation",
     "dataset",
     "transformation",
 ]

dlt/common/data_writers/writers.py

@@ -190,16 +190,9 @@ def write_header(self, columns_schema: TTableSchemaColumns) -> None:
     def write_data(self, items: Sequence[TDataItem]) -> None:
         super().write_data(items)
         for item in items:
-            dialect = item.dialect or (self._caps.sqlglot_dialect if self._caps else None)
-            query = item.query
-            parsed_query = sqlglot.parse_one(query, read=dialect)
-
-            # Ensure the parsed query is a SELECT statement
-            if not isinstance(parsed_query, sqlglot.exp.Select):
-                raise ValueError("Only SELECT statements are allowed to write model files.")
-
-            normalized_query = parsed_query.sql(dialect=dialect)
-            self._f.write("dialect: " + (dialect or "") + "\n" + normalized_query + "\n")
+            dialect = item.query_dialect()
+            query = item.to_sql()
+            self._f.write("dialect: " + (dialect or "") + "\n" + query + "\n")
 
     @classmethod
     def writer_spec(cls) -> FileWriterSpec:

dlt/common/destination/capabilities.py

@@ -3,7 +3,6 @@
     Any,
     Callable,
     ClassVar,
-    Iterable,
     Literal,
     Optional,
     Sequence,
@@ -12,6 +11,7 @@
     Protocol,
     Type,
 )
+from dlt.common.libs.sqlglot import TSqlGlotDialect
 from dlt.common.data_types import TDataType
 from dlt.common.destination.configuration import ParquetFormatConfiguration
 from dlt.common.exceptions import TerminalValueError
@@ -203,7 +203,7 @@ class DestinationCapabilitiesContext(ContainerInjectableContext):
     enforces_nulls_on_alter: bool = True
     """Tells if destination enforces null constraints when adding NOT NULL columns to existing tables"""
 
-    sqlglot_dialect: Optional[str] = None
+    sqlglot_dialect: Optional[TSqlGlotDialect] = None
     """The SQL dialect used by sqlglot to transpile a query to match the destination syntax."""
 
     parquet_format: Optional[ParquetFormatConfiguration] = None