Merge pull request #215 from databrickslabs/issue_211

Imporvements to lakehouse app #211

Author: ravi-databricks

docs/content/app/_index.md

@@ -0,0 +1,67 @@
+---
+title: "App"
+date: 2025-08-29T14:50:11-04:00
+weight: 63
+draft: false
+---
+
+### Initial Setup
+
+**Q1. Do I need to run an initial setup before using the DLT-META App?**
+Yes. Before using the DLT-META App, you must click the Setup button to create the required dlt-meta environment. This initializes the app and enables you to onboard or manage Lakeflow Declarative Pipelines.
+
+### Features and Capabilities
+
+**Q2. What are the main features of the DLT-META App?**
+The DLT-META App provides several key capabilities:
+- Onboard new Lakeflow Declarative Pipeline through an interactive interface
+- Deploy and manage pipelines directly in the app
+- Run demo flows to explore example pipelines and usage patterns
+- Use the command-line interface (CLI) to automate operations
+
+### Access and Permissions
+
+**Q3. Who can access and use the DLT-META App?**
+Only authenticated Databricks workspace users with appropriate permissions can access and use the app:
+- You need `CAN_USE` permission to run the app
+- You need `CAN_MANAGE` permission to administer it
+- The app can be shared within your workspace or account
+- Every user must log in with their Databricks account credentials
+
+### Resource Access
+
+**Q4. How does catalog and schema access work in the DLT-META App?**
+By default, the app uses a dedicated Service Principal (SP) for all data and resource access:
+- The Service Principal needs explicit permissions (`USE CATALOG`, `USE SCHEMA`, `SELECT`) on all Unity Catalog resources
+- User abilities depend on the Service Principal's access, regardless of URL
+- Optional On-Behalf-Of (OBO) mode uses individual user permissions
+
+### Troubleshooting
+
+**Q5. How should I resolve access errors or permission issues?**
+If you experience access-related errors:
+- Verify Service Principal permissions in Unity Catalog
+- Check app attachments to warehouses and secrets
+- Review recent administrative changes
+- Check audit logs for permission denials
+- Contact your Databricks workspace administrator if needed
+
+### Security and Isolation
+
+**Q6. How is security and isolation managed?**
+The DLT-META App provides enterprise-grade security:
+- Runs in a multi-tenant platform with strong isolation
+- Uses dedicated, isolated serverless compute
+- Restricts sharing to authenticated users only
+- Logs all sharing and permission events
+- No public or anonymous access allowed
+
+### Best Practices
+
+**Q7. What are the security best practices?**
+Follow these guidelines for secure operation:
+- Use minimum required permissions (principle of least privilege)
+- Monitor audit logs regularly
+- Restrict to trusted application code
+- Enable OBO mode carefully
+- Maintain regular security reviews

docs/content/faq/app_faq.md

@@ -0,0 +1,117 @@
+---
+title: "DLT-META Lakehouse App"
+date: 2021-08-04T14:25:26-04:00
+weight: 9
+draft: false
+---
+
+# DLT-META Lakehouse App Setup
+
+## Prerequisites
+
+### System Requirements
+- Python 3.8.0 or higher
+- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html) (latest version, e.g., 0.244.0)
+- Configured workspace access
+
+### Initial Setup
+1. Authenticate with Databricks:
+   ```commandline
+   databricks auth login --host WORKSPACE_HOST
+   ```
+
+2. Setup Python Environment:
+   ```commandline
+   git clone https://github.com/databrickslabs/dlt-meta.git
+   cd dlt-meta
+   python -m venv .venv
+   source .venv/bin/activate
+   pip install databricks-sdk
+   ```
+
+## Deployment Options
+
+### Deploy to Databricks
+
+1. Create Custom App:
+   ```commandline
+   databricks apps create demo-dltmeta
+   ```
+   > Note: Wait for command completion (a few minutes)
+
+2. Setup App Code:
+   ```commandline
+   cd dlt-meta/lakehouse_app
+   
+   # Replace testapp with your preferred folder name
+   databricks sync . /Workspace/Users/<user1.user2>@databricks.com/testapp
+   
+   # Deploy the app
+   databricks apps deploy demo-dltmeta --source-code-path /Workspace/Users/<user1.user2>@databricks.com/testapp
+   ```
+
+3. Access the App:
+   - Open URL from step 1 log, or
+   - Navigate: Databricks Web UI → New → App → Back to App → Search your app name
+
+### Run Locally
+
+1. Setup Environment:
+   ```commandline
+   cd dlt-meta/lakehouse_app
+   pip install -r requirements.txt
+   ```
+
+2. Configure Databricks:
+   ```commandline
+   databricks configure --host <your databricks host url> --token <your token>
+   ```
+
+3. Start App:
+   ```commandline
+   python App.py
+   ```
+   Access at: http://127.0.0.1:5000
+
+## Using DLT-META App
+
+### App User Setup
+![App User Example](/images/app_cli.png)
+
+The app creates a dedicated user account that:
+- Handles onboarding, deployment, and demo execution
+- Requires specific permissions for UC catalogs and schemas
+- Example username format: "app-40zbx9_demo-dltmeta"
+
+### Getting Started
+
+1. Initial Setup:
+   - Launch app in browser
+   - Click "Setup dlt-meta project environment"
+   - This initializes the environment for onboarding and deployment
+
+2. Pipeline Management:
+   - Use "UI" tab to onboard and deploy pipelines
+   - Configure pipelines according to your requirements
+
+   **Onboarding Pipeline:**
+   ![Onboarding UI](/images/app_onboarding.png)
+   *Pipeline onboarding interface for configuring new data pipelines*
+
+   **Deploying Pipeline:**
+   ![Deploy UI](/images/app_deploy_pipeline.png)
+   *Pipeline deployment interface for managing and deploying pipelines*
+
+3. Demo Access:
+   - Available demos can be found under "Demo" tab
+   - Run pre-configured demo pipelines to explore features
+
+   ![App Demo](/images/app_run_demos.png)
+   *Demo interface showing available example pipelines*
+
+4. Command Line Interface:
+   - Access CLI features under the "CLI" tab
+   - Execute commands directly from the web interface
+
+   ![CLI UI](/images/app_cli.png)
+   *CLI interface for command-line operations*

docs/content/getting_started/app.md

@@ -7,7 +7,6 @@
 import logging
 import errno
 import re
-# Use pty to create a pseudo-terminal for better interactive support
 import pty
 import select
 import fcntl
@@ -16,7 +15,6 @@
 import signal
 import json
 
-# Configure logging
 logging.basicConfig(level=logging.INFO,
                     format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
                     handlers=[logging.FileHandler("dlt-meta-app.log"),
@@ -227,15 +225,15 @@ def start_command():
         if 'PYTHONPATH' not in os.environ or not os.path.isdir(os.environ.get('PYTHONPATH', '')):
             commands = [
                 "pip install databricks-cli",
-                # "git clone https://github.com/databrickslabs/dlt-meta.git",
-                "git clone https://github.com/dattawalake/dlt-meta.git",
+                "git clone https://github.com/databrickslabs/dlt-meta.git",
                 f"python -m venv {current_directory}/dlt-meta/.venv",
                 f"export HOME={current_directory}",
                 "cd dlt-meta",
                 "source .venv/bin/activate",
                 f"export PYTHONPATH={current_directory}/dlt-meta/",
                 "pwd",
                 "pip install databricks-sdk",
+                "pip install PyYAML",
             ]
             print("Start setting up dlt-meta environment")
             for c in commands:
@@ -322,6 +320,7 @@ def handle_onboard_form():
         "silver_schema": request.form.get('silver_schema', 'dltmeta_silver_7b4e981029b843c799bf61a0a121b3ca'),
         "dlt_meta_layer": request.form.get('dlt_meta_layer', '1'),
         "bronze_table": request.form.get('bronze_table', 'bronze_dataflowspec'),
+        "silver_table": request.form.get('silver_table', 'silver_dataflowspec'),
         "overwrite": "1" if request.form.get('overwrite') == "1" else "0",
         "version": request.form.get('version', 'v1'),
         "environment": request.form.get('environment', 'prod'),
@@ -375,26 +374,67 @@ def handle_deploy_form():
 def run_demo():
     code_to_run = request.json.get('demo_name', '')
     print(f"processing demo for :{request.json}")
-    current_directory = os.environ['PYTHONPATH']  # os.getcwd()
+    current_directory = os.environ['PYTHONPATH']
     demo_dict = {"demo_cloudfiles": "demo/launch_af_cloudfiles_demo.py",
                  "demo_acf": "demo/launch_acfs_demo.py",
                  "demo_silverfanout": "demo/launch_silver_fanout_demo.py",
-                 "demo_dias": "demo/launch_dais_demo.py"
+                 "demo_dias": "demo/launch_dais_demo.py",
+                 "demo_dlt_sink": "demo/launch_dlt_sink_demo.py",
+                 "demo_dabs": "demo/generate_dabs_resources.py"
                  }
     demo_file = demo_dict.get(code_to_run, None)
     uc_name = request.json.get('uc_name', '')
-    result = subprocess.run(f"python {current_directory}/{demo_file} --uc_catalog_name {uc_name} --profile DEFAULT",
-                            shell=True,
-                            capture_output=True,
-                            text=True
-                            )
+
+    if code_to_run == 'demo_dabs':
+
+        # Step 1: Generate Databricks resources
+        subprocess.run(f"python {current_directory}/{demo_file} --uc_catalog_name {uc_name} "
+                       f"--source=cloudfiles --profile DEFAULT",
+                       shell=True,
+                       capture_output=True,
+                       text=True
+                       )
+
+        # Step 2: Change working directory to demo/dabs for all next commands
+        subprocess.run("databricks bundle validate --profile=DEFAULT", cwd=f"{current_directory}/demo/dabs",
+                       shell=True,
+                       capture_output=True,
+                       text=True)
+
+        # Step 4: Deploy the bundle
+        subprocess.run("databricks bundle deploy --target dev --profile=DEFAULT",
+                       cwd=f"{current_directory}/demo/dabs", shell=True,
+                       capture_output=True,
+                       text=True)
+
+        # Step 5: Run 'onboard_people' task
+        rs1 = subprocess.run("databricks bundle run onboard_people -t dev --profile=DEFAULT",
+                             cwd=f"{current_directory}/demo/dabs", shell=True,
+                             capture_output=True,
+                             text=True)
+        print(f"onboarding completed: {rs1.stdout}")
+        # Step 6: Run 'execute_pipelines_people' task
+        result = subprocess.run("databricks bundle run execute_pipelines_people -t dev --profile=DEFAULT",
+                                cwd=f"{current_directory}/demo/dabs",
+                                shell=True,
+                                capture_output=True,
+                                text=True
+                                )
+        print(f"execution of pipeline completed: {result.stdout}")
+    else:
+        result = subprocess.run(f"python {current_directory}/{demo_file} --uc_catalog_name {uc_name} "
+                                f"--profile DEFAULT",
+                                shell=True,
+                                capture_output=True,
+                                text=True
+                                )
     return extract_command_output(result)
 
 
 def extract_command_output(result):
     stdout = result.stdout
     job_id_match = re.search(r"job_id=(\d+) | pipeline=(\d+)", stdout)
-    url_match = re.search(r"url=(https?://[^\s]+)", stdout)
+    url_match = re.search(r"(https?://[^\s]+)", stdout)
 
     job_id = job_id_match.group(1) or job_id_match.group(2) if job_id_match else None
     job_url = url_match.group(1) if url_match else None

docs/static/images/app_cli.png

@@ -655,6 +655,11 @@ <h2 class='step-heading'>Step 1 : Onboarding</h2>
                                     <input id="bronze_table" name="bronze_table" placeholder="Enter bronze table name"
                                            type="text" value="bronze_dataflowspec">
                                 </div>
+                                <div class="form-group">
+                                    <label>Provide silver dataflow spec table name:</label>
+                                    <input id="silver_table" name="silver_table" placeholder="Enter silver table name"
+                                           type="text" value="silver_dataflowspec">
+                                </div>
                                 <div class="form-group">
                                     <label>Overwrite dataflow spec?</label>
                                     <div class="radio-group">
@@ -841,6 +846,8 @@ <h3 class='step-heading'>Available Demos</h3>
                     <button class="command-button2" data-command="demo_acf">Demo Apply Changes Snapshot</button>
                     <button class="command-button2" data-command="demo_silverfanout">Demo Silver fanout</button>
                     <button class="command-button2" data-command="demo_dias">Demo Dias</button>
+                    <button class="command-button2" data-command="demo_dlt_sink">Demo Sink</button>
+
 
                 </div>
 
@@ -956,7 +963,7 @@ <h5 class="modal-title">Please wait...</h5>
                     const modalContent = `
                                 <div class="modal-content">
                                 <h3 class='step-heading'>${data.modal_content.title}</h3>
-                                <p>Job ID: ${data.modal_content.job_id}</p>
+                                ${data.modal_content.job_id ? `<p>Job ID: ${data.modal_content.job_id}</p>`:""}
                                 
                                 <p><a href="${url}" target="_blank">Open Job in Databricks</a></p >
                                
@@ -994,7 +1001,8 @@ <h3 class='step-heading'>${data.modal_content.title}</h3>
                     const modalContent = `
                                 <div class="modal-content">
                                 <h3 class='step-heading'>${data.modal_content.title}</h3>
-                                <p>Job ID: ${data.modal_content.job_id}</p>
+                    ${data.modal_content.job_id ? `<p>Job ID: ${data.modal_content.job_id}</p>
+                `:""}
                                 
                                 <p><a href="${url}" target="_blank">Open Job in Databricks</a></p >
                                
@@ -1067,7 +1075,7 @@ <h3 class='step-heading'>${data.modal_content.title}</h3>
                             const modalContent = `
                                 <div class="modal-content">
                                 <h3 class='step-heading'>${data.modal_content.title}</h3>
-                                <p>Job ID: ${data.modal_content.job_id}</p>
+                                 ${data.modal_content.job_id ? `<p>Job ID: ${data.modal_content.job_id}</p>`:""}
                                 
                                 <p><a href="${url}" target="_blank">Open Job in Databricks</a></p >
                                

docs/static/images/app_deploy_pipeline.png

@@ -638,7 +638,7 @@ def _load_onboard_config_ui(self, form_data) -> OnboardCommand:
                 onboard_cmd_dict["bronze_dataflowspec_path"] = f'{self._install_folder()}/bronze_dataflow_specs'
 
         if onboard_cmd_dict["onboard_layer"] == "silver" or onboard_cmd_dict["onboard_layer"] == "bronze_silver":
-            onboard_cmd_dict["silver_dataflowspec_table"] = 'silver_dataflowspec'  # Not in form, using default
+            onboard_cmd_dict["silver_dataflowspec_table"] = form_data.get('silver_table', 'silver_dataflowspec')
             if not onboard_cmd_dict["uc_enabled"]:
                 onboard_cmd_dict["silver_dataflowspec_path"] = f'{self._install_folder()}/silver_dataflow_specs'