Merge pull request #27 from databrickslabs/feature/dlt-meta-uc-cli

-Fixed integration tests for non-uc flows

Author: ravi-databricks

.gitignore

@@ -154,4 +154,6 @@ deployment-merged.yaml
 integration-tests/conf/dlt-meta/onboarding.json
 
 .databricks
-.databricks-login.json
+.databricks-login.json
+demo/conf/onboarding.json
+integration_tests/conf/onboarding.json

README.md

@@ -71,42 +71,47 @@ Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting
 ### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal
 - ``` git clone dlt-meta ```
 - ``` cd dlt-meta ```
+- ``` python -m venv .venv ```
+- ```source .venv/bin/activate ```
+- ``` pip install databricks ```
+- ``` pip install databricks-sdk ```
 - ```databricks labs dlt-meta onboard``` 
-- - Above command will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder
- ```Provide onboarding file path (default: demo/conf/onboarding.template): 
-    Provide onboarding files local directory (default: demo/): 
-    Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): 
-    Provide databricks runtime version (default: 14.2.x-scala2.12): 
-    Run onboarding with unity catalog enabled?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide unity catalog name: ravi_dlt_meta_uc
-    Provide dlt meta schema name (default: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead): 
-    Provide dlt meta bronze layer schema name (default: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb): 
-    Provide dlt meta silver layer schema name (default: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f): 
-    Provide dlt meta layer
-    [0] bronze
-    [1] bronze_silver
-    [2] silver
-    Enter a number between 0 and 2: 1
-    Provide bronze dataflow spec table name (default: bronze_dataflowspec): 
-    Provide silver dataflow spec table name (default: silver_dataflowspec): 
-    Overwrite dataflow spec?
-    [0] False
-    [1] True
-    Enter a number between 0 and 1: 1
-    Provide dataflow spec version (default: v1): 
-    Provide environment name (default: prod): prod
-    Provide import author name (default: ravi.gawai): 
-    Provide cloud provider name
-    [0] aws
-    [1] azure
-    [2] gcp
-    Enter a number between 0 and 2: 0
-    Do you want to update ws paths, catalog, schema details to your onboarding file?
-    [0] False
-    [1] True
+- - Above command will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder.
+
+```     Provide onboarding file path (default: demo/conf/onboarding.template): 
+        Provide onboarding files local directory (default: demo/): 
+        Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): 
+        Provide databricks runtime version (default: 14.2.x-scala2.12): 
+        Run onboarding with unity catalog enabled?
+        [0] False
+        [1] True
+        Enter a number between 0 and 1: 1
+        Provide unity catalog name: ravi_dlt_meta_uc
+        Provide dlt meta schema name (default: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead): 
+        Provide dlt meta bronze layer schema name (default: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb): 
+        Provide dlt meta silver layer schema name (default: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f): 
+        Provide dlt meta layer
+        [0] bronze
+        [1] bronze_silver
+        [2] silver
+        Enter a number between 0 and 2: 1
+        Provide bronze dataflow spec table name (default: bronze_dataflowspec): 
+        Provide silver dataflow spec table name (default: silver_dataflowspec): 
+        Overwrite dataflow spec?
+        [0] False
+        [1] True
+        Enter a number between 0 and 1: 1
+        Provide dataflow spec version (default: v1): 
+        Provide environment name (default: prod): prod
+        Provide import author name (default: ravi.gawai): 
+        Provide cloud provider name
+        [0] aws
+        [1] azure
+        [2] gcp
+        Enter a number between 0 and 2: 0
+        Do you want to update ws paths, catalog, schema details to your onboarding file?
+        [0] False
+        [1] True
 ```
 - Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs
 - Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command

demo/README.md

@@ -27,12 +27,12 @@ This Demo launches Bronze and Silver DLT pipleines with following activities:
     - cloud_provider_name : aws or azure or gcp
     - db_version : Databricks Runtime Version
     - dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines
-    - you can provide --profile=databricks_profile name in case you already have databricks cli otherwise command prompt will ask host and token
+    - you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token.
 
-    6a. Databricks Workspace URL:
-        - Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
+    - - 6a. Databricks Workspace URL:
+    - - Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
 
-    6b. Token:
+    - - 6b. Token:
         - In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.
 
         - On the Access tokens tab, click Generate new token.
@@ -65,12 +65,12 @@ This demo will launch auto generated tables(100s) inside single bronze and silve
     - cloud_provider_name : aws or azure or gcp
     - db_version : Databricks Runtime Version
     - dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines
-    - you can provide --profile=databricks_profile name in case you already have databricks cli otherwise command prompt will ask host and token
+    - you can provide `--profile=databricks_profile name` in case you already have databricks cli otherwise command prompt will ask host and token
 
-    6a. Databricks Workspace URL:
+    - - 6a. Databricks Workspace URL:
         - Enter your workspace URL, with the format https://<instance-name>.cloud.databricks.com. To get your workspace URL, see Workspace instance names, URLs, and IDs.
 
-    6b. Token:
+    - - 6b. Token:
         - In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the drop down.
 
         - On the Access tokens tab, click Generate new token.

demo/conf/onboarding.template

@@ -12,6 +12,7 @@
       },
       "bronze_database_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_table": "customers",
+      "bronze_table_path_prod": "{dbfs_path}/data/bronze/customers",
       "bronze_reader_options": {
          "cloudFiles.format": "csv",
          "cloudFiles.rescuedDataColumn": "_rescued_data",
@@ -20,8 +21,10 @@
       "bronze_data_quality_expectations_json_prod": "{dbfs_path}/demo/conf/dqe/customers.json",
       "bronze_database_quarantine_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_quarantine_table": "customers_quarantine",
+      "bronze_quarantine_table_path_prod": "{dbfs_path}/data/bronze/customers_quarantine",
       "silver_database_prod": "{uc_catalog_name}.{silver_schema}",
       "silver_table": "customers",
+      "silver_table_path_prod": "{dbfs_path}/data/silver/customers",
       "silver_cdc_apply_changes": {
          "keys": [
             "customer_id"
@@ -50,6 +53,7 @@
       },
       "bronze_database_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_table": "transactions",
+      "bronze_table_path_prod": "{dbfs_path}/data/bronze/transactions",     
       "bronze_reader_options": {
          "cloudFiles.format": "csv",
          "cloudFiles.rescuedDataColumn": "_rescued_data",
@@ -61,6 +65,7 @@
       "bronze_quarantine_table_path_prod": "{dbfs_path}/demo/resources/data/bronze/transactions_quarantine",
       "silver_database_prod": "{uc_catalog_name}.{silver_schema}",
       "silver_table": "transactions",
+      "silver_table_path_prod": "{dbfs_path}/data/silver/transactions",
       "silver_cdc_apply_changes": {
          "keys": [
             "transaction_id"
@@ -90,6 +95,7 @@
       },
       "bronze_database_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_table": "products",
+      "bronze_table_path_prod": "{dbfs_path}/data/bronze/products",
       "bronze_reader_options": {
          "cloudFiles.format": "csv",
          "cloudFiles.rescuedDataColumn": "_rescued_data",
@@ -102,6 +108,7 @@
       "bronze_quarantine_table_path_prod": "{dbfs_path}/demo/resources/data/bronze/products_quarantine",
       "silver_database_prod": "{uc_catalog_name}.{silver_schema}",
       "silver_table": "products",
+      "silver_table_path_prod": "{dbfs_path}/data/silver/products",
       "silver_cdc_apply_changes": {
          "keys": [
             "product_id"
@@ -131,6 +138,7 @@
       },
       "bronze_database_prod": "{uc_catalog_name}.{bronze_schema}",
       "bronze_table": "stores",
+      "bronze_table_path_prod": "{dbfs_path}/data/bronze/stores",
       "bronze_reader_options": {
          "cloudFiles.format": "csv",
          "cloudFiles.rescuedDataColumn": "_rescued_data",
@@ -143,6 +151,7 @@
       "bronze_quarantine_table_path_prod": "{dbfs_path}/demo/resources/data/bronze/stores_quarantine",
       "silver_database_prod": "{uc_catalog_name}.{silver_schema}",
       "silver_table": "stores",
+      "silver_table_path_prod": "{dbfs_path}/data/silver/stores",
       "silver_cdc_apply_changes": {
          "keys": [
             "store_id"

demo/launch_dais_demo.py

@@ -1,5 +1,6 @@
 import uuid
 from databricks.sdk.service import jobs
+from src.install import WorkspaceInstaller
 from integration_tests.run_integration_tests import (
     DLTMETARunner,
     DLTMetaRunnerConf,
@@ -21,6 +22,7 @@ class DLTMETADAISDemo(DLTMETARunner):
     def __init__(self, args, ws, base_dir):
         self.args = args
         self.ws = ws
+        self.wsi = WorkspaceInstaller(ws)
         self.base_dir = base_dir
 
     def init_runner_conf(self) -> DLTMetaRunnerConf:

demo/launch_techsummit_demo.py

@@ -27,6 +27,7 @@
 from databricks.sdk.service.catalog import VolumeType, SchemasAPI
 from databricks.sdk.service.workspace import ImportFormat
 from dataclasses import dataclass
+from src.install import WorkspaceInstaller
 from integration_tests.run_integration_tests import (
     DLTMETARunner,
     DLTMetaRunnerConf,
@@ -65,6 +66,7 @@ class DLTMETATechSummitDemo(DLTMETARunner):
     def __init__(self, args, ws, base_dir):
         self.args = args
         self.ws = ws
+        self.wsi = WorkspaceInstaller(ws)
         self.base_dir = base_dir
 
     def init_runner_conf(self) -> TechsummitRunnerConf:

docs/content/getting_started/additionals.md

@@ -11,35 +11,40 @@ draft: false
 
 2. Goto to DLT-META directory
 
-3. If you already have datatbricks CLI installed with profile as given [here](https://docs.databricks.com/en/dev-tools/cli/profiles.html), you can skip above export step and provide --profile=<your databricks profile name> option while running command 4.
+3. Set python environment variable into terminal
+    ```
+        export PYTHONPATH=<<local dlt-meta path>>
+    ```
 
-4. Run integration test against cloudfile or eventhub or kafka using below options:
-    4a. Run the command for cloudfiles ```python integration-tests/run-integration-test.py  --cloud_provider_name=aws --dbr_version=11.3.x-scala2.12 --source=cloudfiles --dbfs_path=dbfs:/tmp/DLT-META/```
+4. If you already have datatbricks CLI installed with profile as given [here](https://docs.databricks.com/en/dev-tools/cli/profiles.html), you can skip above export step and provide `--profile=<your databricks profile name>` option while running command in step:4
 
-    4b. Run the command for eventhub ```python integration-tests/run-integration-test.py --cloud_provider_name=azure --dbr_version=11.3.x-scala2.12 --source=eventhub --dbfs_path=dbfs:/tmp/DLT-META/ --eventhub_name=iot --eventhub_secrets_scope_name=eventhubs_creds --eventhub_namespace=int_test-standard --eventhub_port=9093 --eventhub_producer_accesskey_name=producer --eventhub_consumer_accesskey_name=consumer```
+5. Run integration test against cloudfile or eventhub or kafka using below options:
+    - 5a. Run the command for cloudfiles ```python integration-tests/run-integration-test.py  --cloud_provider_name=aws --dbr_version=11.3.x-scala2.12 --source=cloudfiles --dbfs_path=dbfs:/tmp/DLT-META/```
 
-    For eventhub integration tests, the following are the prerequisites:
-    1. Needs eventhub instance running
-    2. Using Databricks CLI, Create databricks secrets scope for eventhub keys
-    3. Using Databricks CLI, Create databricks secrets to store producer and consumer keys using the scope created in step 2 
+    - 5b. Run the command for eventhub ```python integration-tests/run-integration-test.py --cloud_provider_name=azure --dbr_version=11.3.x-scala2.12 --source=eventhub --dbfs_path=dbfs:/tmp/DLT-META/ --eventhub_name=iot --eventhub_secrets_scope_name=eventhubs_creds --eventhub_namespace=int_test-standard --eventhub_port=9093 --eventhub_producer_accesskey_name=producer --eventhub_consumer_accesskey_name=consumer```
 
-    Following are the mandatory arguments for running EventHubs integration test
-    1. Provide your eventhub topic : --eventhub_name
-    2. Provide eventhub namespace : --eventhub_namespace
-    3. Provide eventhub port : --eventhub_port
-    4. Provide databricks secret scope name : --eventhub_secrets_scope_name
-    5. Provide eventhub producer access key name : --eventhub_producer_accesskey_name
-    6. Provide eventhub access key name : --eventhub_consumer_accesskey_name
+    - - For eventhub integration tests, the following are the prerequisites:
+        1. Needs eventhub instance running
+        2. Using Databricks CLI, Create databricks secrets scope for eventhub keys
+        3. Using Databricks CLI, Create databricks secrets to store producer and consumer keys using the scope created in step 2 
 
+    - - Following are the mandatory arguments for running EventHubs integration test
+        1. Provide your eventhub topic : --eventhub_name
+        2. Provide eventhub namespace : --eventhub_namespace
+        3. Provide eventhub port : --eventhub_port
+        4. Provide databricks secret scope name : --eventhub_secrets_scope_name
+        5. Provide eventhub producer access key name : --eventhub_producer_accesskey_name
+        6. Provide eventhub access key name : --eventhub_consumer_accesskey_name
 
-    4c. Run the command for kafka ```python3 integration-tests/run-integration-test.py --cloud_provider_name=aws --dbr_version=11.3.x-scala2.12 --source=kafka --dbfs_path=dbfs:/tmp/DLT-META/ --kafka_topic_name=dlt-meta-integration-test --kafka_broker=host:9092```
 
-    For kafka integration tests, the following are the prerequisites:
-    1. Needs kafka instance running
+    - 5c. Run the command for kafka ```python3 integration-tests/run-integration-test.py --cloud_provider_name=aws --dbr_version=11.3.x-scala2.12 --source=kafka --dbfs_path=dbfs:/tmp/DLT-META/ --kafka_topic_name=dlt-meta-integration-test --kafka_broker=host:9092```
 
-    Following are the mandatory arguments for running EventHubs integration test
-    1. Provide your kafka topic name : --kafka_topic_name
-    2. Provide kafka_broker : --kafka_broker
+    - - For kafka integration tests, the following are the prerequisites:
+        1. Needs kafka instance running
+
+    - - Following are the mandatory arguments for running EventHubs integration test
+        1. Provide your kafka topic name : --kafka_topic_name
+        2. Provide kafka_broker : --kafka_broker
 
 6. Once finished integration output file will be copied locally to 
 ```integration-test-output_<run_id>.txt```