Update the default-python template according to Lakeflow conventions (#3712)

## Changes

This updates the `default-python` template according to the latest
Lakeflow conventions as established in
#3671. Notably, the new template
moves away from the use of notebooks for pipeline source code.

The new layout looks as follows when the user selects they want both the
sample job and the sample pipeline:

`📁 resources`
`├── sample_job.job.yml`
`└── sample_etl.pipeline.yml`
`📁 src`
`├── 📁 my_project` — shared source code for use in jobs and/or pipelines
`│   ├── __init__.py`
`│   └── main.py`
`└── 📁 my_project_etl` — source code for the sample_etl pipeline
`    ├── __init__.py`
`    ├── 📁 transformations`
`    │   ├── __init__.py`
`    │   ├── sample_zones_my_project.py`
`    │   └── sample_trips_my_project.py`
`    ├── 📁 explorations` — exploratory notebooks
`    │   ├── __init__.py`
`    │   └── sample_exploration.ipynb`
`    └── README.md`
`📁 tests` — unit tests
`📁 fixtures` — fixtures (these can now be used with
[`load_fixture`](https://github.com/databricks/cli/blob/af524bb993eaffe059d65f93854d544a162fc6ef/acceptance/bundle/templates/default-python/serverless/output/my_default_python/fixtures/.gitkeep))
`databricks.yml`
`pyproject.toml`
`README.md`

The template prompts have been updated to cater to this structure.
Notably, they include a new prompt to manage the catalog and schema used
by the template. These settings are propagated to both the job and the
pipeline:

```
Welcome to the default Python template for Databricks Asset Bundles!

Answer the following questions to customize your project.
You can always change your configuration in the databricks.yml file later.

Note that https://e2-dogfood.staging.cloud.databricks.com is used for initialization.
(For information on how to change your profile, see https://docs.databricks.com/dev-tools/cli/profiles.html.)

Unique name for this project [my_project]: my_project
Include a Lakeflow job that runs a notebook: yes
Include an ETL pipeline: yes
Include a sample Python package that builds into a wheel file: yes
Use serverless compute: yes
Default catalog for any tables created by this project [main]: main
Use a personal schema for each user working on this project.
(This is recommended. Your personal schema will be 'main.lennart_kats'.): yes

✨ Your new project has been created in the 'my_project' directory!

To get started, refer to the project README.md file and the documentation at https://docs.databricks.com/dev-tools/bundles/index.html.
```

## Testing

* Standard unit testing, acceptance testing
* AI excercised templates and with all permutations of options,
deploying/testing/running/inspecting the result
* Bug bash of the original `lakeflow-pipelines` template from
#3671

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Julia Crawford (Databricks) <[email protected]>

Author: 3 people

NEXT_CHANGELOG.md

@@ -9,6 +9,7 @@
 ### Dependency updates
 
 ### Bundles
+* Updated the default-python template to follow the Lakeflow conventions: pipelines as source files, pyproject.toml ([#3712](https://github.com/databricks/cli/pull/3712)).
 * Fix a permissions bug adding second IS\_OWNER and causing "The job must have exactly one owner." error. Introduced in 0.274.0. ([#3850](https://github.com/databricks/cli/pull/3850))
 
 ### API Changes

acceptance/bundle/templates/default-python/classic/out.compare-vs-serverless.diff

@@ -1,68 +1,68 @@
 --- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/databricks.yml
 +++ output/my_default_python/databricks.yml
-@@ -25,4 +25,11 @@
-       host: [DATABRICKS_URL]
-
+@@ -34,4 +34,6 @@
+       catalog: hive_metastore
+       schema: ${workspace.current_user.short_name}
 +    presets:
-+      # Set dynamic_version: true on all artifacts of type "whl".
-+      # This makes "bundle deploy" add a timestamp to wheel's version before uploading,
-+      # new wheel takes over the previous installation even if actual wheel version is unchanged.
-+      # See https://docs.databricks.com/aws/en/dev-tools/bundles/settings
 +      artifacts_dynamic_version: true
-+
    prod:
      mode: production
---- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/resources/my_default_python.job.yml
-+++ output/my_default_python/resources/my_default_python.job.yml
-@@ -17,4 +17,5 @@
-       tasks:
-         - task_key: notebook_task
-+          job_cluster_key: job_cluster
+--- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/resources/my_default_python_etl.pipeline.yml
++++ output/my_default_python/resources/my_default_python_etl.pipeline.yml
+@@ -5,8 +5,7 @@
+     my_default_python_etl:
+       name: my_default_python_etl
+-      # Catalog is required for serverless compute
+-      catalog: main
++      ## Specify the 'catalog' field to configure this pipeline to make use of Unity Catalog:
++      # catalog: ${var.catalog}
+       schema: ${var.schema}
+-      serverless: true
+       root_path: "../src/my_default_python_etl"
+
+--- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/resources/sample_job.job.yml
++++ output/my_default_python/resources/sample_job.job.yml
+@@ -26,4 +26,10 @@
            notebook_task:
-             notebook_path: ../src/notebook.ipynb
-@@ -29,17 +30,21 @@
+             notebook_path: ../src/sample_notebook.ipynb
++          job_cluster_key: job_cluster
++          libraries:
++            # By default we just include the .whl file generated for the my_default_python package.
++            # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
++            # for more information on how to add other libraries.
++            - whl: ../dist/*.whl
+         - task_key: python_wheel_task
            depends_on:
-             - task_key: refresh_pipeline
+@@ -37,5 +43,10 @@
+               - "--schema"
+               - "${var.schema}"
 -          environment_key: default
 +          job_cluster_key: job_cluster
-           python_wheel_task:
-             package_name: my_default_python
-             entry_point: main
 +          libraries:
 +            # By default we just include the .whl file generated for the my_default_python package.
 +            # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
 +            # for more information on how to add other libraries.
 +            - whl: ../dist/*.whl
+         - task_key: refresh_pipeline
+           depends_on:
+@@ -44,11 +55,11 @@
+             pipeline_id: ${resources.pipelines.my_default_python_etl.id}
 
--      # A list of task execution environment specifications that can be referenced by tasks of this job.
 -      environments:
 -        - environment_key: default
--
--          # Full documentation of this spec can be found at:
--          # https://docs.databricks.com/api/workspace/jobs/create#environments-spec
 -          spec:
 -            environment_version: "2"
 -            dependencies:
+-              # By default we just include the .whl file generated for the my_default_python package.
+-              # See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
+-              # for more information on how to add other libraries.
 -              - ../dist/*.whl
 +      job_clusters:
 +        - job_cluster_key: job_cluster
 +          new_cluster:
-+            spark_version: 15.4.x-scala2.12
++            spark_version: 16.4.x-scala2.12
 +            node_type_id: [NODE_TYPE_ID]
 +            data_security_mode: SINGLE_USER
 +            autoscale:
 +              min_workers: 1
 +              max_workers: 4
---- [TESTROOT]/bundle/templates/default-python/classic/../serverless/output/my_default_python/resources/my_default_python.pipeline.yml
-+++ output/my_default_python/resources/my_default_python.pipeline.yml
-@@ -4,8 +4,7 @@
-     my_default_python_pipeline:
-       name: my_default_python_pipeline
--      ## Catalog is required for serverless compute
--      catalog: main
-+      ## Specify the 'catalog' field to configure this pipeline to make use of Unity Catalog:
-+      # catalog: catalog_name
-       schema: my_default_python_${bundle.target}
--      serverless: true
-       libraries:
-         - notebook:

acceptance/bundle/templates/default-python/classic/out.plan_dev.direct.json

@@ -1,10 +1,10 @@
 {
   "plan": {
-    "resources.jobs.my_default_python_job": {
+    "resources.jobs.sample_job": {
       "depends_on": [
         {
-          "node": "resources.pipelines.my_default_python_pipeline",
-          "label": "${resources.pipelines.my_default_python_pipeline.id}"
+          "node": "resources.pipelines.my_default_python_etl",
+          "label": "${resources.pipelines.my_default_python_etl.id}"
         }
       ],
       "action": "create",
@@ -27,43 +27,64 @@
                 "data_security_mode": "SINGLE_USER",
                 "node_type_id": "[NODE_TYPE_ID]",
                 "num_workers": 0,
-                "spark_version": "15.4.x-scala2.12"
+                "spark_version": "16.4.x-scala2.12"
               }
             }
           ],
           "max_concurrent_runs": 4,
-          "name": "[dev [USERNAME]] my_default_python_job",
+          "name": "[dev [USERNAME]] sample_job",
+          "parameters": [
+            {
+              "default": "hive_metastore",
+              "name": "catalog"
+            },
+            {
+              "default": "[USERNAME]",
+              "name": "schema"
+            }
+          ],
           "queue": {
             "enabled": true
           },
           "tags": {
             "dev": "[USERNAME]"
           },
           "tasks": [
+            {
+              "job_cluster_key": "job_cluster",
+              "libraries": [
+                {
+                  "whl": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
+                }
+              ],
+              "notebook_task": {
+                "notebook_path": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src/sample_notebook"
+              },
+              "task_key": "notebook_task"
+            },
             {
               "depends_on": [
                 {
-                  "task_key": "refresh_pipeline"
+                  "task_key": "notebook_task"
                 }
               ],
               "job_cluster_key": "job_cluster",
               "libraries": [
                 {
-                  "whl": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS]-py3-none-any.whl"
+                  "whl": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/artifacts/.internal/my_default_python-0.0.1+[UNIX_TIME_NANOS][0]-py3-none-any.whl"
                 }
               ],
               "python_wheel_task": {
                 "entry_point": "main",
-                "package_name": "my_default_python"
-              },
-              "task_key": "main_task"
-            },
-            {
-              "job_cluster_key": "job_cluster",
-              "notebook_task": {
-                "notebook_path": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src/notebook"
+                "package_name": "my_default_python",
+                "parameters": [
+                  "--catalog",
+                  "hive_metastore",
+                  "--schema",
+                  "[USERNAME]"
+                ]
               },
-              "task_key": "notebook_task"
+              "task_key": "python_wheel_task"
             },
             {
               "depends_on": [
@@ -72,7 +93,7 @@
                 }
               ],
               "pipeline_task": {
-                "pipeline_id": "${resources.pipelines.my_default_python_pipeline.id}"
+                "pipeline_id": "${resources.pipelines.my_default_python_etl.id}"
               },
               "task_key": "refresh_pipeline"
             }
@@ -86,33 +107,36 @@
           }
         },
         "vars": {
-          "tasks[2].pipeline_task.pipeline_id": "${resources.pipelines.my_default_python_pipeline.id}"
+          "tasks[2].pipeline_task.pipeline_id": "${resources.pipelines.my_default_python_etl.id}"
         }
       }
     },
-    "resources.pipelines.my_default_python_pipeline": {
+    "resources.pipelines.my_default_python_etl": {
       "action": "create",
       "new_state": {
         "config": {
           "channel": "CURRENT",
-          "configuration": {
-            "bundle.sourcePath": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src"
-          },
           "deployment": {
             "kind": "BUNDLE",
             "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/state/metadata.json"
           },
           "development": true,
           "edition": "ADVANCED",
+          "environment": {
+            "dependencies": [
+              "--editable /Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files"
+            ]
+          },
           "libraries": [
             {
-              "notebook": {
-                "path": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src/pipeline"
+              "glob": {
+                "include": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src/my_default_python_etl/transformations/**"
               }
             }
           ],
-          "name": "[dev [USERNAME]] my_default_python_pipeline",
-          "schema": "my_default_python_dev",
+          "name": "[dev [USERNAME]] my_default_python_etl",
+          "root_path": "/Workspace/Users/[USERNAME]/.bundle/my_default_python/dev/files/src/my_default_python_etl",
+          "schema": "[USERNAME]",
           "tags": {
             "dev": "[USERNAME]"
           }

acceptance/bundle/templates/default-python/classic/out.plan_dev.terraform.json

@@ -1,9 +1,9 @@
 {
   "plan": {
-    "resources.jobs.my_default_python_job": {
+    "resources.jobs.sample_job": {
       "action": "create"
     },
-    "resources.pipelines.my_default_python_pipeline": {
+    "resources.pipelines.my_default_python_etl": {
       "action": "create"
     }
   }