Adding new demo's under doc site

Author: ravi-databricks

docs/content/demo/Append_FLOW_CF.md

@@ -21,15 +21,26 @@ This demo will perform following tasks:
     databricks auth login --host WORKSPACE_HOST
     ```
 
-3. ```commandline
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. Navigate to project directory:
+    ```commandline
     cd dlt-meta
     ```
 
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
@@ -38,7 +49,8 @@ This demo will perform following tasks:
     export PYTHONPATH=$dlt_meta_home
     ```
 
-6. ```commandline
+7. Run the command:
+    ```commandline
     python demo/launch_af_cloudfiles_demo.py --cloud_provider_name=aws --dbr_version=15.3.x-scala2.12 --dbfs_path=dbfs:/tmp/DLT-META/demo/ --uc_catalog_name=dlt_meta_uc
     ```
 

docs/content/demo/Append_FLOW_EH.md

@@ -18,21 +18,32 @@ draft: false
     databricks auth login --host WORKSPACE_HOST
     ```
 
-3. ```commandline
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. Navigate to project directory:
+    ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
     ```commandline
     export PYTHONPATH=$dlt_meta_home
     ```
-6. Eventhub
+7. Configure Eventhub
 - Needs eventhub instance running
 - Need two eventhub topics first for main feed (eventhub_name) and second for append flow feed (eventhub_name_append_flow)
 - Create databricks secrets scope for eventhub keys
@@ -61,7 +72,8 @@ draft: false
     - eventhub_secrets_scope_name: Databricks secret scope name e.g. eventhubs_dltmeta_creds
     - eventhub_port: Eventhub port
 
-7. ```commandline 
+8. Run the command:
+    ```commandline 
     python demo/launch_af_eventhub_demo.py --cloud_provider_name=aws --uc_catalog_name=dlt_meta_uc --eventhub_name=dltmeta_demo --eventhub_name_append_flow=dltmeta_demo_af --eventhub_secrets_scope_name=dltmeta_eventhub_creds --eventhub_namespace=dltmeta --eventhub_port=9093 --eventhub_producer_accesskey_name=RootManageSharedAccessKey --eventhub_consumer_accesskey_name=RootManageSharedAccessKey --eventhub_accesskey_secret_name=RootManageSharedAccessKey
     ```
 

docs/content/demo/Apply_Changes_From_Snapshot.md

@@ -26,21 +26,33 @@ draft: false
     databricks auth login --host WORKSPACE_HOST
     ```
     
-3. ```commandline
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. Navigate to project directory:
+    ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
     ```commandline
     export PYTHONPATH=$dlt_meta_home
 
-6. ```commandline
+7. Run the command:
+    ```commandline
     python demo/launch_acfs_demo.py --uc_catalog_name=<<uc catalog name>>
     ```
     - uc_catalog_name : Unity catalog name

docs/content/demo/DAB.md

@@ -0,0 +1,98 @@
+---
+title: "DAB Demo"
+date: 2024-02-26T14:25:26-04:00
+weight: 28
+draft: false
+---
+
+### DAB Demo
+
+## Overview
+This demo showcases how to use Databricks Asset Bundles (DABs) with DLT-Meta:
+
+This demo will perform following steps:
+- Create dlt-meta schema's for dataflowspec and bronze/silver layer
+- Upload necessary resources to unity catalog volume
+- Create DAB files with catalog, schema, file locations populated
+- Deploy DAB to databricks workspace
+- Run onboarding using DAB commands
+- Run Bronze/Silver Pipelines using DAB commands
+- Demo examples will showcase fan-out pattern in silver layer
+- Demo example will show case custom transformations for bronze/silver layers
+- Adding custom columns and metadata to Bronze tables
+- Implementing SCD Type 1 to Silver tables
+- Applying expectations to filter data in Silver tables
+
+### Steps:
+1. Launch Command Prompt
+
+2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
+    - Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:
+    
+    ```commandline
+    databricks auth login --host WORKSPACE_HOST
+    ```
+
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
+    git clone https://github.com/databrickslabs/dlt-meta.git 
+    ```
+
+5. Navigate to project directory:
+    ```commandline
+    cd dlt-meta
+    ```
+
+6. Set python environment variable into terminal:
+    ```commandline
+    dlt_meta_home=$(pwd)
+    export PYTHONPATH=$dlt_meta_home
+    ```
+
+7. Generate DAB resources and set up schemas:
+    This command will:
+    - Generate DAB configuration files
+    - Create DLT-Meta schemas
+    - Upload necessary files to volumes
+    ```commandline
+    python demo/generate_dabs_resources.py --source=cloudfiles --uc_catalog_name=<your_catalog_name> --profile=<your_profile>
+    ```
+    > Note: If you don't specify `--profile`, you'll be prompted for your Databricks workspace URL and access token.
+
+8. Deploy and run the DAB bundle:
+    - Navigate to the DAB directory:
+    ```commandline
+    cd demo/dabs
+    ```
+
+    - Validate the bundle configuration:
+    ```commandline
+    databricks bundle validate --profile=<your_profile>
+    ```
+
+    - Deploy the bundle to dev environment:
+    ```commandline
+    databricks bundle deploy --target dev --profile=<your_profile>
+    ```
+
+    - Run the onboarding job:
+    ```commandline
+    databricks bundle run onboard_people -t dev --profile=<your_profile>
+    ```
+
+    - Execute the pipelines:
+    ```commandline
+    databricks bundle run execute_pipelines_people -t dev --profile=<your_profile>
+    ```
+
+![dab_onboarding_job.png](/images/dab_onboarding_job.png)
+![dab_dlt_pipelines.png](/images/dab_dlt_pipelines.png)

docs/content/demo/DAIS.md

@@ -23,23 +23,35 @@ This demo showcases DLT-META's capabilities of creating Bronze and Silver DLT pi
     databricks auth login --host WORKSPACE_HOST
     ```
 
-3. ```commandline
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. Navigate to project directory:
+    ```commandline
     cd dlt-meta
     ```
 
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
     ```commandline
     export PYTHONPATH=$dlt_meta_home
     ```
 
-6. ```commandline 
+7. Run the command:
+    ```commandline 
     python demo/launch_dais_demo.py --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=<<>>
     ```
     - uc_catalog_name : unit catalog name

docs/content/demo/DLT_Sink.md

@@ -0,0 +1,74 @@
+---
+title: "Lakeflow Declarative Pipelines Sink Demo"
+date: 2024-02-26T14:25:26-04:00
+weight: 27
+draft: false
+---
+
+### Lakeflow Declarative Pipelines Sink Demo
+This demo will perform following steps:
+- Showcase onboarding process for dlt writing to external sink pattern
+- Run onboarding for the bronze iot events
+- Publish test events to kafka topic
+- Run Bronze Lakeflow Declarative Pipelines which will read from kafka source topic and write to:
+  - Events delta table into UC
+  - Create quarantine table as per data quality expectations
+  - Writes to external kafka topics
+  - Writes to external dbfs location as external delta sink
+
+### Steps:
+1. Launch Command Prompt
+
+2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
+    - Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:
+    
+    ```commandline
+    databricks auth login --host WORKSPACE_HOST
+    ```
+
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
+    git clone https://github.com/databrickslabs/dlt-meta.git 
+    ```
+
+5. Navigate to project directory:
+    ```commandline
+    cd dlt-meta
+    ```
+
+6. Set python environment variable into terminal:
+    ```commandline
+    dlt_meta_home=$(pwd)
+    export PYTHONPATH=$dlt_meta_home
+    ```
+
+7. Configure Kafka (Optional):
+    If you are using secrets for kafka, create databricks secrets scope for source and sink kafka:
+    ```commandline 
+    databricks secrets create-scope <<n>>
+    ```
+    ```commandline
+    databricks secrets put-secret --json '{
+        "scope": "<<n>>",
+        "key": "<<keyname>>",
+        "string_value": "<<value>>"
+    }'
+    ```
+
+8. Run the command:
+    ```commandline
+    python demo/launch_dlt_sink_demo.py --uc_catalog_name=<<uc_catalog_name>> --source=kafka --kafka_source_topic=<<kafka source topic name>> --kafka_sink_topic=<<kafka sink topic name>> --kafka_source_servers_secrets_scope_name=<<kafka source servers secret name>> --kafka_source_servers_secrets_scope_key=<<kafka source server secret scope key name>> --kafka_sink_servers_secret_scope_name=<<kafka sink server secret scope key name>> --kafka_sink_servers_secret_scope_key=<<kafka sink servers secret scope key name>> --profile=<<DEFAULT>>
+    ```
+
+![dlt_demo_sink.png](/images/dlt_demo_sink.png)
+![dlt_delta_sink.png](/images/dlt_delta_sink.png)
+![dlt_kafka_sink.png](/images/dlt_kafka_sink.png)

docs/content/demo/Silver_Fanout.md

@@ -23,31 +23,43 @@ draft: false
     databricks auth login --host WORKSPACE_HOST
     ```
     
-3. ```commandline
+3. Install Python package requirements:
+    ```commandline
+    # Core requirements
+    pip install "PyYAML>=6.0" setuptools databricks-sdk
+
+    # Development requirements
+    pip install flake8==6.0 delta-spark==3.0.0 pytest>=7.0.0 coverage>=7.0.0 pyspark==3.5.5
+    ```
+
+4. Clone dlt-meta:
+    ```commandline
     git clone https://github.com/databrickslabs/dlt-meta.git 
     ```
 
-4. ```commandline
+5. Navigate to project directory:
+    ```commandline
     cd dlt-meta
     ```
-5. Set python environment variable into terminal
+6. Set python environment variable into terminal
     ```commandline
     dlt_meta_home=$(pwd)
     ```
     ```commandline
     export PYTHONPATH=$dlt_meta_home
 
-6. ```commandline
+7. Run the command:
+    ```commandline
     python demo/launch_silver_fanout_demo.py --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws
     ```
     - uc_catalog_name : aws or azure
     - cloud_provider_name : aws or azure
     - you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.
 
-    - - 6a. Databricks Workspace URL:
-    - - Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
+    a. Databricks Workspace URL:
+       Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
 
-    - - 6b. Token:
+    b. Token:
         - In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.
 
         - On the Access tokens tab, click Generate new token.