Merge pull request #220 from databrickslabs/issue_219

Replacing DLT with Lakeflow Declarative Pipelines

Author: ravi-databricks

README.md

@@ -32,7 +32,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 #### Generic Lakeflow Declarative Pipeline
 
 - Apply appropriate readers based on input metadata
-- Apply data quality rules with DLT expectations
+- Apply data quality rules with Lakeflow Declarative Pipeline expectations
 - Apply CDC apply changes if specified in metadata
 - Builds Lakeflow Declarative Pipeline graph based on input/output metadata
 - Launch Lakeflow Declarative Pipeline pipeline
@@ -45,7 +45,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 
 ![DLT-META Stages](./docs/static/images/dlt-meta_stages.png)
 
-## DLT-META DLT Features support
+## DLT-META Lakeflow Declarative Pipeline Features support
 | Features  | DLT-META Support |
 | ------------- | ------------- |
 | Input data sources  | Autoloader, Delta, Eventhub, Kafka, snapshot  |
@@ -58,7 +58,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 | [append_flow](https://docs.databricks.com/en/delta-live-tables/flows.html#use-append-flow-to-write-to-a-streaming-table-from-multiple-source-streams) API support | Bronze layer|
 | Liquid cluster support | Bronze, Bronze Quarantine, Silver tables|
 | [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
-| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using default publishing mode |
 | [create_sink](https://docs.databricks.com/aws/en/dlt-ref/dlt-python-ref-sink) API support |Supported formats:```external delta table , kafka``` Bronze, Silver layers|
 | [Databricks Asset Bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/) | Supported
 | [DLT-META UI](https://github.com/databrickslabs/dlt-meta/tree/main/lakehouse_app#dlt-meta-lakehouse-app-setup) | Uses Databricks Lakehouse DLT-META App
@@ -166,7 +166,7 @@ The command will prompt you to provide pipeline configuration details.
 ![deployingDLTMeta_bronze_silver.gif](docs/static/images/deployingDLTMeta_bronze_silver.gif)
 
 Above deploy cli command will:
-1. Deploy Lakeflow Declarative pipeline with dlt-meta configuration like ```layer```, ```group```, ```dataflowSpec table details``` etc to your databricks workspace
+1. Deploy Lakeflow Declarative Pipeline with dlt-meta configuration like ```layer```, ```group```, ```dataflowSpec table details``` etc to your databricks workspace
 2. Display message: ```dlt-meta pipeline={pipeline_id} created and launched with update_id={pipeline_update_id}, url=https://{databricks workspace url}/#joblist/pipelines/{pipeline_id}```
 3. Pipline URL will automatically open in your defaul browser.
 

demo/README.md

@@ -13,7 +13,7 @@
 ## [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA)
 This Demo launches Bronze and Silver pipelines with following activities:
 - Customer and Transactions feeds for initial load
-- Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes.
+- Adds new feeds Product and Stores to existing Bronze and Silver Lakeflow Declarative Pipeline pipelines with metadata changes.
 - Runs Bronze and Silver pipeline for incremental load for CDC events
 
 ### Steps:

demo/notebooks/afam_cloudfiles_runners/validate.py

@@ -23,7 +23,7 @@
 }
 
 
-log_list.append("Validating DLT Bronze and Silver Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...")
 for table, counts in UC_TABLES.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")
     cnt = query.collect()[0].cnt

demo/notebooks/afam_eventhub_runners/validate.py

@@ -21,7 +21,7 @@
     f"{uc_catalog_name}.{bronze_schema}.bronze_{run_id}_iot_quarantine": 2
 }
 
-log_list.append("Validating DLT EVenthub Bronze Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline EVenthub Bronze Table Counts...")
 tables = UC_TABLES if uc_enabled else NON_UC_TABLES
 for table, counts in tables.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")

demo/notebooks/dlt_sink_runners/validate.py

@@ -15,7 +15,7 @@
     f"{uc_catalog_name}.{bronze_schema}.bronze_{run_id}_iot_quarantine": 2,
 }
 
-log_list.append("Validating DLT EVenthub Bronze Table Counts...")
+log_list.append("Validating Lakeflow Declarative Pipeline EVenthub Bronze Table Counts...")
 for table, counts in TABLES.items():
     query = spark.sql(f"SELECT count(*) as cnt FROM {table}")
     cnt = query.collect()[0].cnt

docs/content/_index.md

@@ -36,7 +36,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
     - Option#2: [Manual Job](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#onboardjob)
     - option#3: [Databricks Notebook](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#option2-databricks-notebook)
 
-- Dataflow DLT Pipeline
+- Dataflow Lakeflow Declarative Pipeline
     - Option#1: [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/#dataflow-dlt-pipeline)
     - Option#2: [DLT-META MANUAL](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#dataflow-dlt-pipeline)
 
@@ -53,7 +53,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 | [append_flow](https://docs.databricks.com/aws/en/dlt-ref/dlt-python-ref-append-flow) API support | Bronze layer|
 | Liquid cluster support | Bronze, Bronze Quarantine, Silver, Silver Quarantine tables|
 | [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
-| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
+| Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using default publishing mode |
 | [DLT Sinks](https://docs.databricks.com/aws/en/dlt/dlt-sinks) | Supported formats:external ```delta table```, ```kafka```.Bronze, Silver layers|
 ## How much does it cost ?
 DLT-META does not have any **direct cost** associated with it other than the cost to run the Databricks Lakeflow Declarative Pipelines 

docs/content/additionals/integration_tests.md

@@ -71,7 +71,7 @@ draft: false
 ,0
 0,Completed Bronze Lakeflow Declarative Pipeline.
 1,Completed Silver Lakeflow Declarative Pipeline.
-2,Validating DLT Bronze and Silver Table Counts...
+2,Validating Lakeflow Declarative Pipeline Bronze and Silver Table Counts...
 3,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions.
 4,Expected: 10002 Actual: 10002. Passed!
 5,Validating Counts for Table bronze_7d1d3ccc9e144a85b07c23110ea50133.transactions_quarantine.

docs/content/demo/Apply_Changes_From_Snapshot.md

@@ -9,11 +9,11 @@ draft: false
   - This demo will perform following steps
     - Showcase onboarding process for apply changes from snapshot pattern
     - Run onboarding for the bronze stores and products tables, which contains data snapshot data in csv files.
-    - Run Bronze DLT to load initial snapshot (LOAD_1.csv)
+    - Run Bronze Lakeflow Declarative Pipeline to load initial snapshot (LOAD_1.csv)
     - Upload incremental snapshot LOAD_2.csv version=2 for stores and product
-    - Run Bronze DLT to load incremental snapshot (LOAD_2.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
+    - Run Bronze Lakeflow Declarative Pipeline to load incremental snapshot (LOAD_2.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
     - Upload incremental snapshot LOAD_3.csv version=3 for stores and product
-    - Run Bronze DLT to load incremental snapshot (LOAD_3.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
+    - Run Bronze Lakeflow Declarative Pipeline to load incremental snapshot (LOAD_3.csv). Stores is scd_type=2 so updated records will expired and added new records with version_number. Products is scd_type=1 so in case records missing for scd_type=1 will be deleted.
 
 
 ### Steps:

docs/content/demo/DAIS.md

@@ -8,10 +8,10 @@ draft: false
 ### DAIS 2023 DEMO:
 #### [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA)
 
-This demo showcases DLT-META's capabilities of creating Bronze and Silver DLT pipelines with initial and incremental mode automatically.
+This demo showcases DLT-META's capabilities of creating Bronze and Silver Lakeflow Declarative Pipeline pipelines with initial and incremental mode automatically.
 - Customer and Transactions feeds for initial load
-- Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes.
-- Runs Bronze and Silver DLT for incremental load for CDC events
+- Adds new feeds Product and Stores to existing Bronze and Silver Lakeflow Declarative Pipeline pipelines with metadata changes.
+- Runs Bronze and Silver Lakeflow Declarative Pipeline for incremental load for CDC events
 
 #### Steps to launch DAIS demo in your Databricks workspace:
 1. Launch Command Prompt

docs/content/demo/Techsummit.md

@@ -6,7 +6,7 @@ draft: false
 ---
 
 ### Databricks Tech Summit FY2024 DEMO:
-This demo will launch auto generated tables(100s) inside single bronze and silver DLT pipeline using dlt-meta.
+This demo will launch auto generated tables(100s) inside single bronze and silver Lakeflow Declarative Pipeline using dlt-meta.
 
 1. Launch Command Prompt