Make DB Connect work out of the box for unit tests with the default-python template

[lennartkats-db]

Changes

This update default-python to make DB Connect work out of the box:

• We now install a conservative 15.x version of DB Connect using uv (14.x is even older and doesn't support serverless)
• We validate that the current DB Connect version is supported by the remote cluster / serverless compute. Note that the VS Code extension also does this, providing a warning when users attempt to run against an older cluster.
• We make sure there's always a valid SparkSession during test execution (even when users write standard Spark / Spark Declarative Pipelines code with SparkSessions.builder.getOrCreate())
• We fall back to serverless compute when the user didn't manually configure a cluster. This also addresses known a current issue with the VS Code extension where it doesn't provide a cluster id when using 'Run' rather than 'Debug' to execute unit tests.
• We validate that test dependencies are installed and report a friendly error if they are not.

Usage

1. Install via bundle init

lennart:demo$ databricks bundle init
Template to use: python-default

Welcome to the default Python template for Databricks Asset Bundles!
Please provide the following details to tailor the template to your preferences.

Unique name for this project [my_project]: my_project
Include a stub (sample) notebook in 'my_project/src': yes
Include a stub (sample) Delta Live Tables pipeline in 'my_project/src': yes
Include a stub (sample) Python package in 'my_project/src': yes
Use serverless compute: yes
Workspace to use (auto-detected, edit in 'my_project/databricks.yml'): https://[workspace].databricks.com

✨ Your new project has been created in the 'my_project' directory!

Please refer to the README.md file for "getting started" instructions.
See also the documentation at https://docs.databricks.com/dev-tools/bundles/index.html.
lennart:demo$ cd my_project

2. Run tests with pytest

lennart:my_project$ pytest
ImportError while loading conftest '/private/tmp/demp/my_project/conftest.py'.
conftest.py:17: in <module>
    raise ImportError(
E   ImportError: Test dependencies not found.
E
E   Run tests using 'uv run pytest'. See http://docs.astral.sh/uv to learn move about uv.

3. Actually, Step 2 was wrong. Run tests using uv. Or from the VS Code extension.

lennart:my_project$ uv run pytest
Using CPython 3.13.3 interpreter at: /opt/homebrew/opt/python@3.13/bin/python3.13
Creating virtual environment at: .venv
   Built my-project @ file:///private/tmp/demo/my_project
Installed 32 packages in 235ms
☁️ no compute specified, falling back to serverless compute
  see https://docs.databricks.com/dev-tools/databricks-connect/cluster-config for manual configuration
== test session starts ==
platform darwin -- Python 3.13.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /private/tmp/demo/my_project
configfile: pyproject.toml
testpaths: tests
collected 1 item

tests/main_test.py .                                                                                                                                                                                         [100%]

== 1 passed in 1.71s ==

Why

Manually setting up testing with DB Connect is still rather hard. With these changes, uv can be used to setup testing automatically.

Tests

• Acceptance tests
• Manual tests from CLI, Cursor, different versions of DB Connect

[eng-dev-ecosystem-bot]

Run: 17401350474

	Env	✅​pass	🔄​flaky	🙈​skip
✅​	aws linux	308		511
✅​	aws windows	309		510
✅​	aws-ucws linux	420		409
✅​	aws-ucws windows	421		408
✅​	azure linux	308		510
✅​	azure windows	309		509
🔄​	azure-ucws linux	417	3	408
✅​	azure-ucws windows	421		407
✅​	gcp linux	307		512
✅​	gcp windows	308		511

Test Name	azure-ucws linux
TestAccept	🔄​flaky
TestAccept/bundle/local_state_staleness	🔄​flaky
TestAccept/bundle/local_state_staleness/DATABRICKS_CLI_DEPLOYMENT=direct-exp	🔄​flaky