Prepare a new "generic" template, use "src layout" for Lakeflow template (#3671)

## Changes

This PR prepares a generic `default` template that I want to use as the
basis for `default-python`, `lakeflow-pipelines`, and (likely)
`default-sql`:
* This template revises and replaces `lakeflow-pipelines`
* The template now uses an "src" layout with job/pipeline environments
pointing to the directory of the pyproject.toml file
* Jobs in this template now pass the `catalog` and `schema` parameters
since we ask a question about catalog/schema in the template

To support the notion of a "generic" template, the template schema
format now supports a `template_dir` argument. This allows us to have
multiple `databricks_template_schema.json` files that point to one
template directory.

Out of scope: this PR does not yet update `default-python`. For early
testing purposes, an early version is available as
`experimental-default-python`. To keep the diff cleaner, I removed
acceptance tests for this template; that's something for a follow-up PR.

## Why
* We want to follow the Lakeflow conventions in templates
* Our templates are hard to maintain and have many inconsistencies. I'd
like to move to one shared template for at least default-python and the
Lakeflow template.

## Tests
* Standard template testing methodology.

<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->

---------

Co-authored-by: Claude <[email protected]>

Author: lennartkats-db

NEXT_CHANGELOG.md

@@ -11,5 +11,6 @@
 ### Dependency updates
 
 ### Bundles
+* Updated the internal lakeflow-pipelines template to use an "src" layout ([#3671](https://github.com/databricks/cli/pull/3671)).
 
 ### API Changes

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/README.md

@@ -1,10 +1,8 @@
 # my_lakeflow_pipelines
 
-The 'my_lakeflow_pipelines' project was generated by using the Lakeflow Pipelines template.
+The 'my_lakeflow_pipelines' project was generated by using the default template.
 
-* `lib/`: Python source code for this project.
-* `lib/shared`: Shared source code across all jobs/pipelines/etc.
-* `resources/lakeflow_pipelines_etl`: Pipeline code and assets for the lakeflow_pipelines_etl pipeline.
+* `src/`: Python source code for this project.
 * `resources/`:  Resource configurations (jobs, pipelines, etc.)
 
 ## Getting started
@@ -46,7 +44,7 @@ with this project. It's also possible to interact with it directly using the CLI
    $ databricks bundle deploy --target prod
    ```
    Note the default template has a includes a job that runs the pipeline every day
-   (defined in resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml). The schedule
+   (defined in resources/sample_job.job.yml). The schedule
    is paused when deploying in development mode (see
    https://docs.databricks.com/dev-tools/bundles/deployment-modes.html).
 

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "my_lakeflow_pipelines"
+version = "0.0.1"
+authors = [{ name = "[USERNAME]" }]
+requires-python = ">=3.10,<=3.13"
+dependencies = [
+    # Any dependencies for jobs and pipelines in this project can be added here
+    # See also https://docs.databricks.com/dev-tools/bundles/library-dependencies
+    #
+    # LIMITATION: for pipelines, dependencies are cached during development;
+    # add dependencies to the 'environment' section of pipeline.yml file instead
+]
+
+[dependency-groups]
+dev = [
+    "pytest",
+    "databricks-dlt",
+    "databricks-connect>=15.4,<15.5",
+]
+
+[project.scripts]
+main = "my_lakeflow_pipelines.main:main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.black]
+line-length = 125

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl.pipeline.yml

@@ -0,0 +1,21 @@
+# The main pipeline for my_lakeflow_pipelines
+
+resources:
+  pipelines:
+    lakeflow_pipelines_etl:
+      name: lakeflow_pipelines_etl
+      ## Catalog is required for serverless compute
+      catalog: ${var.catalog}
+      schema: ${var.schema}
+      serverless: true
+      root_path: "../src/lakeflow_pipelines_etl"
+
+      libraries:
+        - glob:
+            include: ../src/lakeflow_pipelines_etl/transformations/**
+
+      environment:
+        dependencies:
+          # We include every dependency defined by pyproject.toml by defining an editable environment
+          # that points to the folder where pyproject.toml is deployed.
+          - --editable ${workspace.file_path}

acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_etl.pipeline.yml

@@ -1,9 +1,9 @@
-# The job that triggers lakeflow_pipelines_etl.
+# A sample job for my_lakeflow_pipelines.
 
 resources:
   jobs:
-    lakeflow_pipelines_job:
-      name: lakeflow_pipelines_job
+    sample_job:
+      name: sample_job
 
       trigger:
         # Run this job every day, exactly one day from the last run; see https://docs.databricks.com/api/workspace/jobs/create#trigger
@@ -15,7 +15,18 @@ resources:
       #  on_failure:
       #    - [email protected]
 
+      parameters:
+        - name: catalog
+          default: ${var.catalog}
+        - name: schema
+          default: ${var.schema}
+
       tasks:
         - task_key: refresh_pipeline
           pipeline_task:
             pipeline_id: ${resources.pipelines.lakeflow_pipelines_etl.id}
+
+      environments:
+        - environment_key: default
+          spec:
+            environment_version: "2"

…lines_etl/lakeflow_pipelines_job.job.yml …w_pipelines/resources/sample_job.job.ymlacceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml renamed to acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/sample_job.job.yml acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/lakeflow_pipelines_etl/lakeflow_pipelines_job.job.yml renamed to acceptance/bundle/templates/lakeflow-pipelines/python/output/my_lakeflow_pipelines/resources/sample_job.job.yml

@@ -12,11 +12,9 @@ This folder defines all source code for the my_lakeflow_pipelines pipeline:
 To get started, go to the `transformations` folder -- most of the relevant source code lives there:
 
 * By convention, every dataset under `transformations` is in a separate file.
-* Take a look at the sample under "sample_trips_my_lakeflow_pipelines.py" to get familiar with the syntax.
+* Take a look at the sample called "sample_trips_my_lakeflow_pipelines.py" to get familiar with the syntax.
   Read more about the syntax at https://docs.databricks.com/dlt/python-ref.html.
-* Use `Run file` to run and preview a single transformation.
-* Use `Run pipeline` to run _all_ transformations in the entire pipeline.
-* Use `+ Add` in the file browser to add a new data set definition.
-* Use `Schedule` to run the pipeline on a schedule!
+* If you're using the workspace UI, use `Run file` to run and preview a single transformation.
+* If you're using the CLI, use `databricks bundle run lakeflow_pipelines_etl --select sample_trips_my_lakeflow_pipelines` to run a single transformation.
 
 For more tutorials and reference material, see https://docs.databricks.com/dlt.