Update the lakeflow-pipelines template according to the latest Lakeflow conventions (#3558)

This adds a new version of the `default-python` template. To enable the
first integration and evaluation of this template, this first pull
request replaces the internal `lakeflow-pipelines` template. Replacing
`default-python` will be the next milestone.

## Changes
* Update `default-python` to follow the Lakeflow conventions
(per-pipeline sources go into `resources/`; shared sources go into
(`lib/`)
* Adapt this template so it can first be used in place of the
`lakeflow-pipelines` template (this required disabling some of the
template questions and components)

## Why
Lakeflow comes with new conventions that emphasize modularity. This PR
represents an initial milestone towards updating the DABs
`default-python` template. `default-sql` will follow in a later
milestone.

## Tests
Existing tests were updated.

<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->

Author: lennartkats-db

acceptance/bundle/templates/lakeflow-pipelines/python/output.txt

@@ -1,12 +1,16 @@
 
 >>> [CLI] bundle init lakeflow-pipelines --config-file ./input.json --output-dir output
-
 Welcome to the template for Lakeflow Declarative Pipelines!
 
+Please answer the below to tailor your project to your preferences.
+You can always change your mind and change your configuration in the databricks.yml file later.
+
+Note that [DATABRICKS_URL] is used for initialization
+(see https://docs.databricks.com/dev-tools/cli/profiles.html for how to change your profile).
 
-Your new project has been created in the 'my_lakeflow_pipelines' directory!
+✨ Your new project has been created in the 'my_lakeflow_pipelines' directory!
 
-Refer to the README.md file for "getting started" instructions!
+Please refer to the README.md file for "getting started" instructions.
 
 >>> [CLI] bundle validate -t dev
 Name: my_lakeflow_pipelines

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/extensions.json

@@ -1,7 +1,7 @@
 {
     "recommendations": [
         "databricks.databricks",
-        "ms-python.vscode-pylance",
-        "redhat.vscode-yaml"
+        "redhat.vscode-yaml",
+        "ms-python.black-formatter"
     ]
 }

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/.vscode/settings.json

@@ -1,19 +1,37 @@
 {
-    "python.analysis.stubPath": ".vscode",
-    "databricks.python.envFile": "${workspaceFolder}/.env",
     "jupyter.interactiveWindow.cellMarker.codeRegex": "^# COMMAND ----------|^# Databricks notebook source|^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
     "jupyter.interactiveWindow.cellMarker.default": "# COMMAND ----------",
     "python.testing.pytestArgs": [
         "."
     ],
-    "python.testing.unittestEnabled": false,
-    "python.testing.pytestEnabled": true,
-    "python.analysis.extraPaths": ["resources/my_lakeflow_pipelines_pipeline"],
     "files.exclude": {
         "**/*.egg-info": true,
         "**/__pycache__": true,
         ".pytest_cache": true,
+        "dist": true,
+    },
+    "files.associations": {
+        "**/.gitkeep": "markdown"
     },
+
+    // Pylance settings (VS Code)
+    // Set typeCheckingMode to "basic" to enable type checking!
+    "python.analysis.typeCheckingMode": "off",
+    "python.analysis.extraPaths": ["src", "lib", "resources"],
+    "python.analysis.diagnosticMode": "workspace",
+    "python.analysis.stubPath": ".vscode",
+
+    // Pyright settings (Cursor)
+    // Set typeCheckingMode to "basic" to enable type checking!
+    "cursorpyright.analysis.typeCheckingMode": "off",
+    "cursorpyright.analysis.extraPaths": ["src", "lib", "resources"],
+    "cursorpyright.analysis.diagnosticMode": "workspace",
+    "cursorpyright.analysis.stubPath": ".vscode",
+
+    // General Python settings
+    "python.defaultInterpreterPath": "./.venv/bin/python",
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true,
     "[python]": {
         "editor.defaultFormatter": "ms-python.black-formatter",
         "editor.formatOnSave": true,

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/README.md

@@ -2,38 +2,53 @@
 
 The 'my_lakeflow_pipelines' project was generated by using the Lakeflow Pipelines template.
 
-## Setup
+* `lib/`: Python source code for this project.
+* `lib/shared`: Shared source code across all jobs/pipelines/etc.
+* `resources/lakeflow_pipelines_etl`: Pipeline code and assets for the lakeflow_pipelines_etl pipeline.
+* `resources/`:  Resource configurations (jobs, pipelines, etc.)
 
-1. Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/databricks-cli.html
+## Getting started
 
-2. Authenticate to your Databricks workspace, if you have not done so already:
-    ```
-    $ databricks auth login
-    ```
+Choose how you want to work on this project:
+
+(a) Directly in your Databricks workspace, see
+    https://docs.databricks.com/dev-tools/bundles/workspace.
 
-3. Optionally, install developer tools such as the Databricks extension for Visual Studio Code from
-   https://docs.databricks.com/dev-tools/vscode-ext.html. Or the PyCharm plugin from
-   https://www.databricks.com/blog/announcing-pycharm-integration-databricks.
+(b) Locally with an IDE like Cursor or VS Code, see
+    https://docs.databricks.com/vscode-ext.
 
+(c) With command line tools, see https://docs.databricks.com/dev-tools/cli/databricks-cli.html
 
-## Deploying resources
+# Using this project using the CLI
 
-1. To deploy a development copy of this project, type:
+The Databricks workspace and IDE extensions provide a graphical interface for working
+with this project. It's also possible to interact with it directly using the CLI:
+
+1. Authenticate to your Databricks workspace, if you have not done so already:
+    ```
+    $ databricks configure
+    ```
+
+2. To deploy a development copy of this project, type:
     ```
     $ databricks bundle deploy --target dev
     ```
     (Note that "dev" is the default target, so the `--target` parameter
     is optional here.)
 
-2. Similarly, to deploy a production copy, type:
-   ```
-   $ databricks bundle deploy --target prod
-   ```
+    This deploys everything that's defined for this project.
+    For example, the default template would deploy a pipeline called
+    `[dev yourname] lakeflow_pipelines_etl` to your workspace.
+    You can find that resource by opening your workpace and clicking on **Jobs & Pipelines**.
 
-3. Use the "summary" comand to review everything that was deployed:
+3. Similarly, to deploy a production copy, type:
    ```
-   $ databricks bundle summary
+   $ databricks bundle deploy --target prod
    ```
+   Note the default template has a includes a job that runs the pipeline every day
+   (defined in resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml). The schedule
+   is paused when deploying in development mode (see
+   https://docs.databricks.com/dev-tools/bundles/deployment-modes.html).
 
 4. To run a job or pipeline, use the "run" command:
    ```

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/databricks.yml

@@ -14,8 +14,6 @@ variables:
     description: The catalog to use
   schema:
     description: The schema to use
-  notifications:
-    description: The email addresses to use for failure notifications
 
 targets:
   dev:
@@ -30,18 +28,15 @@ targets:
     variables:
       catalog: main
       schema: ${workspace.current_user.short_name}
-      notifications: []
-
   prod:
     mode: production
     workspace:
       host: [DATABRICKS_URL]
       # We explicitly deploy to /Workspace/Users/[USERNAME] to make sure we only have a single copy.
       root_path: /Workspace/Users/[USERNAME]/.bundle/${bundle.name}/${bundle.target}
+    variables:
+      catalog: main
+      schema: prod
     permissions:
       - user_name: [USERNAME]
         level: CAN_MANAGE
-    variables:
-      catalog: main
-      schema: default
-      notifications: [[USERNAME]]

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/lib/shared/__init__.py

@@ -0,0 +1,7 @@
+from databricks.sdk.runtime import spark
+from pyspark.sql import DataFrame
+
+
+def find_all_taxis() -> DataFrame:
+    """Find all taxi data."""
+    return spark.read.table("samples.nyctaxi.trips")

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/lib/shared/taxis.py

@@ -4,5 +4,7 @@ dist/
 __pycache__/
 *.egg-info
 .venv/
+scratch/**
+!scratch/README.md
 **/explorations/**
 **/!explorations/README.md

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/out.gitignore

@@ -0,0 +1 @@
+This folder is reserved for Databricks Asset Bundles resource definitions.

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/.gitkeep

@@ -1,11 +1,11 @@
-# my_lakeflow_pipelines_pipeline
+# my_lakeflow_pipelines
 
-This folder defines all source code for the my_lakeflow_pipelines_pipeline pipeline:
+This folder defines all source code for the my_lakeflow_pipelines pipeline:
 
-- `explorations`: Ad-hoc notebooks used to explore the data processed by this pipeline.
-- `transformations`: All dataset definitions and transformations.
-- `utilities` (optional): Utility functions and Python modules used in this pipeline.
-- `data_sources` (optional): View definitions describing the source data for this pipeline.
+- `explorations/`: Ad-hoc notebooks used to explore the data processed by this pipeline.
+- `transformations/`: All dataset definitions and transformations.
+- `utilities/` (optional): Utility functions and Python modules used in this pipeline.
+- `data_sources/` (optional): View definitions describing the source data for this pipeline.
 
 ## Getting Started