exp: @dlt.transformation ergonomics

[zilto]

Description

See tests/transformations/test_ergonomics.py for the main contribution. The user-facing API is streamlined and compatible with eager and lazy execution. This is a tentative to simplify the @dlt.transformation mechanisms.

Goals:

• Make it easy to execute functions decorated with @dlt.transformation outside pipeline.run()
• Make transformations yield instead of return
• Remove the need for multiple ReadableDataset implementations.
• Remove the need for multiple ReadableRelation implementations.
• Remove the need for ReadableRelation (still debating this one)
• Make typing easier and more reliable
• The transformation function shouldn't depend on dlt.Dataset if it doesn't access it (e.g., returns a constant SQL string)
• Limit custom __call__, __getitem__, __getattr__ usage
• SQLGlot schema should be inferred against a dataset, not the relation (it needs to know of all tables). We could have a function that takes in a dataset

  def sqlglot_schema(
    sql: str | exp.Select | SqlModel,
    dataset: dlt.Dataset | None = None,
    discover: bool = False,  # use the SQLGlot DDL PR to retrieve non-dlt tables too
  ) -> sqlglot.schema.Schema:
     ...

Notes

This is a draft PR. Goal is to showcase the API (via tests) and overall design.

Remaining todos:

• Deprecate unused code branch
• Enable backwards compatibility for dataset access, or add deprecation warnings
• Move code from dlt/transformations/ergonomics.py to the right files and make tests pass

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	e45204c
🔍 Latest deploy log	https://app.netlify.com/projects/dlt-hub-docs/deploys/684b76a978b7f20008f5a434

[zilto]

Closing in favor of #2718