|
1 | | -# DLT-META Lakehouse App Setup |
2 | | - |
3 | | -Make sure you have installed/upgraded the latest Databricks CLI version (e.g., 0.244.0) and configured workspace access where the app is being deployed. |
4 | | - |
5 | | -## Create App and Attach Source to Databricks Apps |
6 | | - |
7 | | -### Step 1: Create a Custom App ("empty") Using the CLI |
8 | | -For example, if the app name is `demo-dltmeta`: |
9 | | -```bash |
10 | | -databricks apps create demo-dltmeta |
11 | | -``` |
12 | | -Wait for the command execution to complete. It will take a few minutes. |
13 | | - |
14 | | -### Step 2: Checkout Project from DLT-META Git Repository |
15 | | -```bash |
16 | | -git clone https://github.com/databrickslabs/dlt-meta.git |
17 | | -``` |
18 | | - |
19 | | -### Step 3: Navigate to the Project Directory |
20 | | -```bash |
21 | | -cd dlt-meta/lakehouse_app |
22 | | -``` |
23 | | - |
24 | | -### Step 4: Sync the DLT-META App Code to Your Workspace Directory |
25 | | -Run the command below to sync the code (replace `testapp` with your desired folder name): |
26 | | -```bash |
27 | | -databricks sync . /Workspace/Users/<user1.user2>@databricks.com/testapp |
28 | | -``` |
29 | | - |
30 | | -### Step 5: Deploy Code to the App Created in Step 1 |
31 | | -```bash |
32 | | -databricks apps deploy demo-dltmeta --source-code-path /Workspace/Users/<user1.user2>@databricks.com/testapp |
33 | | -``` |
34 | | - |
35 | | -### Step 6: Open the App in the Browser |
36 | | -- Open the URL from the Step 1 log, or |
37 | | -- Go to the Databricks web page, click **New > App**, click back on **App**, search for your app name, and click on the URL to open the app in the browser. |
38 | | - |
39 | | ---- |
40 | | - |
41 | | -## Run the App Locally |
42 | | - |
43 | | -### Step 1: Checkout Project from DLT-META Git Repository |
44 | | -```bash |
45 | | -git clone https://github.com/databrickslabs/dlt-meta.git |
46 | | -``` |
47 | | - |
48 | | -### Step 2: Navigate to the Project Directory |
49 | | -```bash |
50 | | -cd dlt-meta/lakehouse_app |
51 | | -``` |
52 | | - |
53 | | -### Step 3: Install the Required Dependencies |
54 | | -```bash |
55 | | -pip install -r requirements.txt |
56 | | -``` |
57 | | - |
58 | | -### Step 4: Configure Databricks |
59 | | -```bash |
60 | | -databricks configure --host <your-databricks-host-url> --token <your-token> |
61 | | -``` |
62 | | - |
63 | | -### Step 5: Run the App |
64 | | -```bash |
65 | | -python App.py |
66 | | -``` |
67 | | - |
68 | | -### Step 6: Access the App |
69 | | -Click on the URL link: [http://127.0.0.1:5000](http://127.0.0.1:5000) |
70 | | - |
71 | | ---- |
72 | | - |
73 | | -## Databricks App Username |
74 | | - |
75 | | -Databricks creates a unique username for each app, which can be found on the Databricks app page. |
76 | | - |
77 | | -### Step 1: Configure the DLT-META Environment |
78 | | -After launching the app in the browser, click the button **"Setup DLT-META Project Environment"** to configure the DLT-META environment on the app's remote instance for onboarding and deployment activities. |
79 | | - |
80 | | -### Step 2: Onboard a DLT Pipeline |
81 | | -Use the **"UI"** tab to onboard and deploy DLT pipelines based on your pipeline configuration. |
82 | | - |
83 | | -### Step 3: Run Available Demos |
84 | | -Navigate to the **"Demo"** tab to run the available demos. |
| 1 | +# DLT-META Lakehouse App |
| 2 | + |
| 3 | +## Prerequisites |
| 4 | + |
| 5 | +### System Requirements |
| 6 | +- Python 3.8.0 or higher |
| 7 | +- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html) (latest version, e.g., 0.244.0) |
| 8 | +- Configured workspace access |
| 9 | + |
| 10 | +### Initial Setup |
| 11 | +1. Authenticate with Databricks: |
| 12 | + ```commandline |
| 13 | + databricks auth login --host WORKSPACE_HOST |
| 14 | + ``` |
| 15 | + |
| 16 | +2. Setup Python Environment: |
| 17 | + ```commandline |
| 18 | + git clone https://github.com/databrickslabs/dlt-meta.git |
| 19 | + cd dlt-meta |
| 20 | + python -m venv .venv |
| 21 | + source .venv/bin/activate |
| 22 | + pip install databricks-sdk |
| 23 | + ``` |
| 24 | + |
| 25 | +## Deployment Options |
| 26 | + |
| 27 | +### Deploy to Databricks |
| 28 | + |
| 29 | +1. Create Custom App: |
| 30 | + ```commandline |
| 31 | + databricks apps create demo-dltmeta |
| 32 | + ``` |
| 33 | + > Note: Wait for command completion (a few minutes) |
| 34 | +
|
| 35 | +2. Setup App Code: |
| 36 | + ```commandline |
| 37 | + cd dlt-meta/lakehouse_app |
| 38 | + |
| 39 | + # Replace testapp with your preferred folder name |
| 40 | + databricks sync . /Workspace/Users/<user1.user2>@databricks.com/testapp |
| 41 | + |
| 42 | + # Deploy the app |
| 43 | + databricks apps deploy demo-dltmeta --source-code-path /Workspace/Users/<user1.user2>@databricks.com/testapp |
| 44 | + ``` |
| 45 | + |
| 46 | +3. Access the App: |
| 47 | + - Open URL from step 1 log, or |
| 48 | + - Navigate: Databricks Web UI → New → App → Back to App → Search your app name |
| 49 | + |
| 50 | +### Run Locally |
| 51 | + |
| 52 | +1. Setup Environment: |
| 53 | + ```commandline |
| 54 | + cd dlt-meta/lakehouse_app |
| 55 | + pip install -r requirements.txt |
| 56 | + ``` |
| 57 | + |
| 58 | +2. Configure Databricks: |
| 59 | + ```commandline |
| 60 | + databricks configure --host <your databricks host url> --token <your token> |
| 61 | + ``` |
| 62 | + |
| 63 | +3. Start App: |
| 64 | + ```commandline |
| 65 | + python App.py |
| 66 | + ``` |
| 67 | + Access at: http://127.0.0.1:5000 |
| 68 | + |
| 69 | +## Using DLT-META App |
| 70 | + |
| 71 | +### App User Setup |
| 72 | + |
| 73 | + |
| 74 | + |
| 75 | +The app creates a dedicated user account that: |
| 76 | +- Handles onboarding, deployment, and demo execution |
| 77 | +- Requires specific permissions for UC catalogs and schemas |
| 78 | +- Example username format: "app-40zbx9_demo-dltmeta" |
| 79 | + |
| 80 | +### Getting Started |
| 81 | + |
| 82 | +1. Initial Setup: |
| 83 | + - Launch app in browser |
| 84 | + - Click "Setup dlt-meta project environment" |
| 85 | + - This initializes the environment for onboarding and deployment |
| 86 | + |
| 87 | +2. Pipeline Management: |
| 88 | + - Use "UI" tab to onboard and deploy pipelines |
| 89 | + - Configure pipelines according to your requirements |
| 90 | + |
| 91 | + **Onboarding Pipeline:** |
| 92 | + |
| 93 | +  |
| 94 | + |
| 95 | + *Pipeline onboarding interface for configuring new data pipelines* |
| 96 | + |
| 97 | + **Deploying Pipeline:** |
| 98 | + |
| 99 | +  |
| 100 | + |
| 101 | + *Pipeline deployment interface for managing and deploying pipelines* |
| 102 | + |
| 103 | +3. Demo Access: |
| 104 | + - Available demos can be found under "Demo" tab |
| 105 | + - Run pre-configured demo pipelines to explore features |
| 106 | + |
| 107 | +  |
| 108 | + |
| 109 | + *Demo interface showing available example pipelines* |
| 110 | + |
| 111 | +4. Command Line Interface: |
| 112 | + - Access CLI features under the "CLI" tab |
| 113 | + - Execute commands directly from the web interface |
| 114 | + |
| 115 | +  |
| 116 | + |
| 117 | + *CLI interface for command-line operations* |
0 commit comments