Merge pull request #207 from databrickslabs/Issue_22

Issue 22

Author: ravi-databricks

.gitignore

@@ -156,5 +156,4 @@ demo/conf/onboarding.json
 integration_tests/conf/onboarding*.json
 demo/conf/onboarding*.json
 integration_test_output*.csv
-databricks.yml
 onboarding_job_details.json

README.md

@@ -60,6 +60,7 @@ In practice, a single generic pipeline reads the Dataflowspec and uses it to orc
 | [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/) |  ```databricks labs dlt-meta onboard```, ```databricks labs dlt-meta deploy``` |
 | Bronze and Silver pipeline chaining | Deploy dlt-meta pipeline with ```layer=bronze_silver``` option using Direct publishing mode |
 | [DLT Sinks](https://docs.databricks.com/aws/en/delta-live-tables/dlt-sinks) |Supported formats:external ```delta table```, ```kafka```.Bronze, Silver layers|
+| [Databricks Asset Bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/) | Supported
 
 ## Getting Started
 
@@ -99,36 +100,47 @@ databricks auth login --host WORKSPACE_HOST
 
 If you want to run existing demo files please follow these steps before running onboard command:
 
-```commandline
+1. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
-```
+    ```
 
-```commandline
+2. Navigate to project directory:
+    ```commandline
     cd dlt-meta
-```
+    ```
 
-```commandline
+3. Create Python virtual environment:
+    ```commandline
     python -m venv .venv
-```
+    ```
 
-```commandline
+4. Activate virtual environment:
+    ```commandline
     source .venv/bin/activate
-```
+    ```
 
-```commandline
-    pip install databricks-sdk
-```
+5. Install required packages:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+    
+    # Development requirements
+    pip install delta-spark==3.0.0 pyspark==3.5.5 pytest>=7.0.0 coverage>=7.0.0
+    
+    # Integration test requirements
+    pip install "typer[all]==0.6.1"
+    ```
 
-```commandline
+6. Set environment variables:
+    ```commandline
     dlt_meta_home=$(pwd)
-```
-
-```commandline
     export PYTHONPATH=$dlt_meta_home
-```
-```commandline
+    ```
+7. Run onboarding command:
+    ```commandline
     databricks labs dlt-meta onboard
-```
+    ```
 ![onboardingDLTMeta.gif](docs/static/images/onboardingDLTMeta.gif)
 
 

demo/README.md

@@ -21,15 +21,22 @@ This Demo launches Bronze and Silver pipelines with following activities:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
 
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -38,7 +45,7 @@ This Demo launches Bronze and Silver pipelines with following activities:
     export PYTHONPATH=$dlt_meta_home
     ```
 
-6. ```commandline
+7. ```commandline
     python demo/launch_dais_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
     ```
     - uc_catalog_name : Unity catalog name
@@ -53,15 +60,21 @@ This demo will launch auto generated tables(100s) inside single bronze and silve
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
 
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -70,7 +83,7 @@ This demo will launch auto generated tables(100s) inside single bronze and silve
     export PYTHONPATH=$dlt_meta_home
     ```
 
-6. ```commandline
+7. ```commandline
     python demo/launch_techsummit_demo.py --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
     ```
     - uc_catalog_name : Unity catalog name
@@ -89,15 +102,21 @@ This demo will perform following tasks:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
 
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -106,7 +125,7 @@ This demo will perform following tasks:
     export PYTHONPATH=$dlt_meta_home
     ```
 
-6. ```commandline
+7. ```commandline
     python demo/launch_af_cloudfiles_demo.py --uc_catalog_name=<<uc catalog name>> --source=cloudfiles --profile=<<DEFAULT>>
     ```
     - uc_catalog_name : Unity Catalog name
@@ -122,14 +141,20 @@ This demo will perform following tasks:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -181,14 +206,20 @@ This demo will perform following tasks:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -198,15 +229,15 @@ This demo will perform following tasks:
 
 6. Run the command 
     ```commandline
-        python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
+    python demo/launch_silver_fanout_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --profile=<<DEFAULT>>
     ```
 
     - you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.
 
-    - - 6a. Databricks Workspace URL:
-    - - Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
+    a. Databricks Workspace URL:
+       Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
 
-    - - 6b. Token:
+    b. Token:
         - In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.
 
         - On the Access tokens tab, click Generate new token.
@@ -241,14 +272,20 @@ This demo will perform following tasks:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -276,14 +313,20 @@ This demo will perform following tasks:
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -316,32 +359,38 @@ This demo will perform following tasks:
 
 ## Overview
 This demo showcases how to use Databricks Asset Bundles (DABs) with DLT-Meta:
-* This demo will perform following steps
-* * Create dlt-meta schema's for dataflowspec and bronze/silver layer
-* * Upload nccessary resources to unity catalog volume
-* * Create DAB files with catalog, schema, file locations populated
-* * Deploy DAB to databricks workspace
-* * Run onboarding usind DAB commands
-* * Run Bronze/Silver Pipelines using DAB commands
-* * Demo examples will showcase fan-out pattern in silver layer
-* * Demo example will show case custom transfomations for bronze/silver layers
-* * Adding custom columns and metadata to Bronze tables
-* * Implementing SCD Type 1 to Silver tables
-* * Applying expectations to filter data in Silver tables
+This demo will perform following steps:
+- Create dlt-meta schema's for dataflowspec and bronze/silver layer
+- Upload nccessary resources to unity catalog volume
+- Create DAB files with catalog, schema, file locations populated
+- Deploy DAB to databricks workspace
+- Run onboarding usind DAB commands
+- Run Bronze/Silver Pipelines using DAB commands
+- Demo examples will showcase fan-out pattern in silver layer
+- Demo example will show case custom transfomations for bronze/silver layers
+- Adding custom columns and metadata to Bronze tables
+- Implementing SCD Type 1 to Silver tables
+- Applying expectations to filter data in Silver tables
 
 ### Steps:
 1. Launch Command Prompt
 
 2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
 
-3. ```commandline
+3. Install Python package requirements:
+   ```commandline
+   pip install "PyYAML>=6.0" setuptools databricks-sdk
+   pip install delta-spark==3.0.0 pyspark==3.5.5
+   ```
+
+4. ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```

demo/dabs/databricks.yml

@@ -0,0 +1,42 @@
+# This is a Databricks asset bundle definition for dbx_dab_dlt_meta_demo_aug_2025.
+# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
+bundle:
+  name: dab_dlt_meta_demo
+  uuid: fb711a7b-ceb7-4054-a49c-cf1d53702692
+
+include:
+  - resources/*.yml
+sync:
+  exclude:
+    - .gitignore
+    - .DS_Store
+    - .vscode
+    - bundle_config_schema.json
+    - .venv/
+    - resources/jobs.template
+
+targets:
+  dev:
+    # The default target uses 'mode: development' to create a development copy.
+    # - Deployed resources get prefixed with '[dev my_user_name]'
+    # - Any job schedules and triggers are paused by default.
+    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.
+    mode: development
+    default: true
+    #SET DATABRICKS_HOST=<your-databricks-workspace-url>
+    # workspace:
+    #   root_path: /Workspace/Shared/${bundle.name}/demo/${bundle.target}
+
+  prod:
+    mode: production
+    workspace:
+      #SET DATABRICKS_HOST=<your-databricks-workspace-url>
+      # host: https://
+      # We explicitly deploy to /Workspace/Users/${bundle.name} to make sure we only have a single copy.
+      root_path: /Workspace/Shared/${bundle.name}/demo/${bundle.target}
+      # Remember to set the correct group name for production deployments.
+    permissions:
+      - group_name: users
+        level: CAN_MANAGE
+      # - user_name: service_principal_id
+      #   level: CAN_MANAGE