Initial implementation of the local cache layer (#3678)

## Changes
This PR introduces a local file-based cache layer for the Databricks CLI
to improve performance of repeated operations.

New libs/cache package:
- Standalone generic function GetOrCompute[T any](ctx, cache,
fingerprint, compute) that works with any type
- File-based cache implementation (fileCache) storing JSON-encoded data
  - SHA256-based fingerprinting for cache keys from any struct
  - Automatic cleanup of expired cache files on initialization
- Fail-open behavior: cache errors never block operations, just trigger
recomputation
- Cache isolation by CLI version:
~/.cache/databricks/<version>/<component>/

Cache Modes:
- Measurement mode (default): Cache disabled, but still measures
potential savings via telemetry
- Enabled mode: Set DATABRICKS_CACHE_ENABLED=true to actually use cached
values

New CLI command:
- databricks cache clear - Removes all cached files across all CLI
versions

Bundle Integration:
- New InitializeCache() mutator to set up cache in bundle initialization
phase
  - PopulateCurrentUser now uses cache for CurrentUser.Me() API call

## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->

This is the first attempt to speed up subsequent `databricks bundle`
commands that a bundle developer runs while developing a bundle

## Tests
<!-- How have you tested the changes? -->
- changed existing acceptance tests to use a dedicated cache folder
- added new acceptance test for overall caching functionality and
telemetry
- added new acceptance test for clearing the cache
- added unit tests for `libs/cache`

<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->

---------

Co-authored-by: Andrew Nester <[email protected]>
Co-authored-by: Andrew Nester <[email protected]>

Author: anton-107

acceptance/acceptance_test.go

@@ -628,6 +628,11 @@ func runTest(t *testing.T,
 		cmd.Env = append(cmd.Env, "GOCOVERDIR="+coverDir)
 	}
 
+	// Set unique cache folder for this test to avoid race conditions between parallel tests
+	// Use test temp directory to avoid polluting user's cache
+	uniqueCacheDir := filepath.Join(t.TempDir(), ".cache")
+	cmd.Env = append(cmd.Env, "DATABRICKS_CACHE_DIR="+uniqueCacheDir)
+
 	for _, key := range utils.SortedKeys(config.Env) {
 		if hasKey(customEnv, key) {
 			// We want EnvMatrix to take precedence.

acceptance/bundle/resource_deps/job_tasks/out.telemetry.direct.txt

@@ -2,6 +2,8 @@ experimental.use_legacy_run_as false
 has_classic_interactive_compute false
 has_classic_job_compute false
 has_serverless_compute true
+local.cache.attempt true
+local.cache.miss true
 presets_name_prefix_is_set false
 python_wheel_wrapper_is_set false
 resref_jobs.tags.* true

acceptance/bundle/resource_deps/job_tasks/out.telemetry.terraform.txt

@@ -2,6 +2,8 @@ experimental.use_legacy_run_as false
 has_classic_interactive_compute false
 has_classic_job_compute false
 has_serverless_compute true
+local.cache.attempt true
+local.cache.miss true
 presets_name_prefix_is_set false
 python_wheel_wrapper_is_set false
 resref_jobs.tags.* true

acceptance/bundle/resource_deps/resources_var/output.txt

@@ -40,6 +40,8 @@ experimental.use_legacy_run_as false
 has_classic_interactive_compute false
 has_classic_job_compute false
 has_serverless_compute false
+local.cache.attempt true
+local.cache.hit true
 presets_name_prefix_is_set true
 python_wheel_wrapper_is_set false
 resref_volumes.catalog_name true

acceptance/bundle/run/scripts/databricks-cli/test.toml

@@ -1,6 +1,9 @@
 RecordRequests = true
 IncludeRequestHeaders = ["Authorization"]
 
+[Env]
+DATABRICKS_CACHE_ENABLED = 'false'
+
 # "client_id:client_secret" in base64 is Y2xpZW50X2lkOmNsaWVudF9zZWNyZXQ=, expect to
 # see this in Authorization header
 [[Repls]]

acceptance/bundle/telemetry/deploy-compute-type/output.txt

@@ -13,6 +13,14 @@ Deployment complete!
 
 >>> cat out.requests.txt
 [
+  {
+    "key": "local.cache.attempt",
+    "value": true
+  },
+  {
+    "key": "local.cache.miss",
+    "value": true
+  },
   {
     "key": "experimental.use_legacy_run_as",
     "value": false
@@ -47,6 +55,14 @@ Deployment complete!
   }
 ]
 [
+  {
+    "key": "local.cache.attempt",
+    "value": true
+  },
+  {
+    "key": "local.cache.hit",
+    "value": true
+  },
   {
     "key": "experimental.use_legacy_run_as",
     "value": false

acceptance/bundle/telemetry/deploy-experimental/output.txt

@@ -12,6 +12,14 @@ Deployment complete!
 >>> cat out.requests.txt
 {
   "bool_values": [
+    {
+      "key": "local.cache.attempt",
+      "value": true
+    },
+    {
+      "key": "local.cache.miss",
+      "value": true
+    },
     {
       "key": "experimental.use_legacy_run_as",
       "value": true

acceptance/bundle/telemetry/deploy-name-prefix/custom/output.txt

@@ -8,6 +8,14 @@ Deployment complete!
 >>> cat out.requests.txt
 {
   "bool_values": [
+    {
+      "key": "local.cache.attempt",
+      "value": true
+    },
+    {
+      "key": "local.cache.miss",
+      "value": true
+    },
     {
       "key": "experimental.use_legacy_run_as",
       "value": false

acceptance/bundle/telemetry/deploy-name-prefix/mode-development/output.txt

@@ -8,6 +8,14 @@ Deployment complete!
 >>> cat out.requests.txt
 {
   "bool_values": [
+    {
+      "key": "local.cache.attempt",
+      "value": true
+    },
+    {
+      "key": "local.cache.miss",
+      "value": true
+    },
     {
       "key": "experimental.use_legacy_run_as",
       "value": false

acceptance/bundle/telemetry/deploy-whl-artifacts/output.txt

@@ -16,6 +16,14 @@ Deployment complete!
 >>> cat out.requests.txt
 {
   "bool_values": [
+    {
+      "key": "local.cache.attempt",
+      "value": true
+    },
+    {
+      "key": "local.cache.miss",
+      "value": true
+    },
     {
       "key": "artifact_build_command_is_set",
       "value": false
@@ -48,6 +56,14 @@ Deployment complete!
 }
 {
   "bool_values": [
+    {
+      "key": "local.cache.attempt",
+      "value": true
+    },
+    {
+      "key": "local.cache.hit",
+      "value": true
+    },
     {
       "key": "artifact_build_command_is_set",
       "value": true