Add default scala template to databricks cli (#3906)

## Changes
<!-- Brief summary of your changes that is easy to understand -->
This adds a new default scala DABs template as a follow up to
databricks/bundle-examples#119
## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
This is to provide an off the shelf template for customers to start
using scala dbconnect to develop scala jobs in Databricks

## Tests
<!-- How have you tested the changes? -->
Manually tested user flow interactively and as job workload for both
standard and serverless
<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->

Author: garlandz-db

NEXT_CHANGELOG.md

@@ -9,5 +9,6 @@
 ### Dependency updates
 
 ### Bundles
+* Add `default-scala` template for Scala projects with SBT build configuration and example code ([#3906](https://github.com/databricks/cli/pull/3906))
 
 ### API Changes

acceptance/bundle/help/bundle-init/output.txt

@@ -6,6 +6,7 @@ TEMPLATE_PATH optionally specifies which template to use. It can be one of the f
 - default-python: The default Python template for Notebooks and Lakeflow
 - default-sql: The default SQL template for .sql files that run with Databricks SQL
 - default-minimal: The minimal template, for advanced users
+- default-scala: The default Scala template for JAR jobs
 - dbt-sql: The dbt SQL template (databricks.com/blog/delivering-cost-effective-data-real-time-dbt-and-databricks)
 - mlops-stacks: The Databricks MLOps Stacks template (github.com/databricks/mlops-stacks)
 - pydabs: A variant of the 'default-python' template that defines resources in Python instead of YAML

acceptance/bundle/templates/default-scala/input.json

@@ -0,0 +1,7 @@
+{
+    "project_name": "my_default_scala",
+    "compute_type": "serverless",
+    "artifacts_dest_path": "/Volumes/test-folder",
+    "default_catalog": "main",
+    "personal_schemas": "yes, use a schema based on the current user name during development"
+}

acceptance/bundle/templates/default-scala/out.test.toml

@@ -0,0 +1,8 @@
+
+>>> [CLI] bundle init default-scala --config-file ./input.json --output-dir output
+
+Welcome to the default-scala template for Databricks Asset Bundles!
+
+A workspace was selected based on your current profile. For information about how to change this, see https://docs.databricks.com/dev-tools/cli/profiles.html.
+workspace_host: [DATABRICKS_URL]
+✨ Successfully initialized template

acceptance/bundle/templates/default-scala/output.txt

@@ -0,0 +1,93 @@
+# my_default_scala
+
+The 'my_default_scala' project was generated by using the default-scala template.
+
+## Getting started
+
+1. Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/install.html. The version must be v0.241.0 or later.
+
+2. Authenticate to your Databricks workspace (if you have not done so already):
+    ```
+    $ databricks configure
+    ```
+
+3. To deploy a development copy of this project, type:
+    ```
+    $ databricks bundle deploy --target dev
+    ```
+    (Note that "dev" is the default target, so the `--target` parameter
+    is optional here.)
+
+    This deploys everything that's defined for this project.
+    For example, the default template would deploy a job called
+    `[dev yourname] my_default_scala_job` to your workspace.
+    You can find that job by opening your workspace and clicking on **Workflows**.
+
+4. Similarly, to deploy a production copy, type:
+   ```
+   $ databricks bundle deploy --target prod
+   ```
+
+5. To run a job, use the "run" command:
+   ```
+   $ databricks bundle run
+   ```
+
+6. Optionally, install developer tools such as the Databricks extension for Visual Studio Code from
+   https://docs.databricks.com/dev-tools/vscode-ext.html.
+
+7. For documentation on the Databricks Asset Bundles format used
+   for this project, and for CI/CD configuration, see
+   https://docs.databricks.com/dev-tools/bundles/index.html.
+
+## Local Devloop
+
+### Prerequisites
+
+Install the following tools:
+
+- [sbt](https://www.scala-sbt.org/) v1.10.2 or later
+- Java 17
+
+### Running via sbt
+
+1. On the terminal, navigate to the project's root directory. This is the directory where the `build.sbt` file is located.
+2. Execute the project's default `Main` class by running `sbt run`.
+
+### IntelliJ setup
+
+Install the latest [IntelliJ IDEA](https://www.jetbrains.com/idea/) IDE, both Community and Professional Editions work.
+Install the [Scala plugin](https://plugins.jetbrains.com/plugin/1347-scala) from the Jetbrains marketplace.
+
+1. Import the current directory in your in IntelliJ where `build.sbt` is located.
+2. Choose the correct Java version in IntelliJ, go to File -> Project Structure -> SDKs.
+
+   Then Run -> Edit Configurations -> Set version to Java 17 from drop
+3. You should now be able to run the code directly in the IDE via the ▶️ option.
+
+#### JVM settings
+
+If you see the following error message,
+
+```
+Failed to initialize MemoryUtil. You must start Java with --add-opens=java.base/java.nio=ALL-UNNAMED
+```
+
+add the following to your JVM settings: `--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`.
+
+See the IntelliJ instructions on how to [configure VM settings for a specific run
+configuration](https://www.jetbrains.com/help/idea/run-debug-configuration-java-application.html#more_options)
+or [configure it everywhere on your IDE](https://www.jetbrains.com/help/ide-services/configure-settings-via-profiles.html).
+
+### Unit tests
+
+The project comes with a sample set of unit tests: `NycTaxiSpec.scala` using the ScalaTest
+framework.
+
+Run the tests either directly in the IntelliJ IDE by clicking ▶️ on the tests, or via sbt
+by running `sbt test`.
+
+## Customizations
+
+### Job configuration
+This project uses serverless compute. No cluster setup is required.

acceptance/bundle/templates/default-scala/output/my_default_scala/README.md

@@ -0,0 +1,29 @@
+// This file is used to build the sbt project with Databricks Connect.
+// This also includes the instructions on how to to create the jar uploaded via databricks bundle
+scalaVersion := "2.13.16"
+
+name := "my_default_scala"
+organization := "com.examples"
+version := "0.1"
+
+libraryDependencies += "com.databricks" %% "databricks-connect" % "17.0.+"
+libraryDependencies += "org.slf4j" % "slf4j-simple" % "2.0.16"
+
+libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.19" % Test
+
+assembly / assemblyOption ~= { _.withIncludeScala(false) }
+assembly / assemblyExcludedJars := {
+  val cp = (assembly / fullClasspath).value
+  cp filter { _.data.getName.matches("scala-.*") } // remove Scala libraries
+}
+
+assemblyMergeStrategy := {
+  case _ => MergeStrategy.preferProject
+}
+
+// to run with new jvm options, a fork is required otherwise it uses same options as sbt process
+fork := true
+javaOptions += "--add-opens=java.base/java.nio=ALL-UNNAMED"
+
+// To ensure logs are written to System.out by default and not System.err
+javaOptions += "-Dorg.slf4j.simpleLogger.logFile=System.out"

acceptance/bundle/templates/default-scala/output/my_default_scala/build.sbt

@@ -0,0 +1,53 @@
+# This is a Databricks asset bundle definition for my_default_scala.
+# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
+bundle:
+  name: my_default_scala
+  uuid: [UUID]
+
+include:
+  - resources/*.yml
+
+variables:
+  catalog:
+    description: The catalog to use
+  schema:
+    description: The schema to use
+
+workspace:
+  host: [DATABRICKS_URL]
+  artifact_path: /Volumes/test-folder/${bundle.name}/${bundle.target}/${workspace.current_user.short_name}
+
+artifacts:
+  default:
+    type: jar
+    build: sbt package && sbt assembly
+    path: .
+    files:
+      - source: ./target/scala-2.13/my_default_scala-assembly-0.1.jar
+
+targets:
+  dev:
+    # The default target uses 'mode: development' to create a development copy.
+    # - Deployed resources get prefixed with '[dev my_user_name]'
+    # - Any job schedules and triggers are paused by default.
+    # See also https://docs.databricks.com/dev-tools/bundles/deployment-modes.html.
+    mode: development
+    default: true
+    workspace:
+      host: [DATABRICKS_URL]
+    variables:
+      catalog: main
+      schema: ${workspace.current_user.short_name}
+
+  prod:
+    mode: production
+    workspace:
+      host: [DATABRICKS_URL]
+      # We explicitly deploy to /Workspace/Users/[USERNAME] to make sure we only have a single copy.
+      root_path: /Workspace/Users/[USERNAME]/.bundle/${bundle.name}/${bundle.target}
+    permissions:
+      - user_name: [USERNAME]
+        level: CAN_MANAGE
+    variables:
+      catalog: main
+      schema: default

acceptance/bundle/templates/default-scala/output/my_default_scala/databricks.yml

@@ -0,0 +1 @@
+.databricks/

acceptance/bundle/templates/default-scala/output/my_default_scala/out.gitignore

@@ -0,0 +1,4 @@
+// The project folder is used to store sbt specific project files
+// This file is used to define the plugins that are used in the sbt project.
+// In particular, this includes the assembly plugin to generate an uber jar.
+addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.0.0")