[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines (#3225)

## Changes
Removed deprecated path fallback mechanism for jobs and pipelines

Effectively, it removes an old deprecated behaviour and only relies on
the new one introduced here #1273

## What's changing?
Assuming there are the following 2 bundle configuration files in your
bundle
- `./resources/my_job.yml` - this is where the job is defined
- `./override.yml` - this is where the notebook path of the job task is
overridden

```
./resources/my_job.yml

resources:
  jobs:
    my_job:
      name: "my job"
      tasks:
        - task_key: notebook_example
          job_cluster_key: default
          notebook_task:
            # if not overridden, this resolves to ./resources/original_notebook.py
            notebook_path: ./original_notebook.py

```

```
./override.yml

targets:
  development:
    resources:
      jobs:
        my_job:
          tasks:
            - task_key: notebook_example
              notebook_task:
                # before this PR: this resolves to <BUNDLE_ROOT>/../src/notebook.py first and if that cannot be done, resolves to <BUNDLE_ROOT>/src/notebook.py
                # with this PR: this resolves only to <BUNDLE_ROOT>/../src/notebook.py
                notebook_path: ../src/notebook.py

```

**Before** this change, on `bundle validate / deploy` the notebook_path
for notebook_example would resolve to `./src/notebook.py` relative to
the bundle root because the relative path was relative to where the job
is defined (`/resources/my_job.yml`)

**After** this change, the notebook_path for notebook_example would
resolve to `../src/notebook.py` relative to the bundle root because now
the path is calculated relative to where it's defined (`./override.yml`)

## Action items
If you are affected by this change, you can do one of the following:
1. **(recommended)** Make necessary configuration changes to have
relative paths defined relative to configuration files they are in, or
2. Pin the CLI version which you use to 0.261.0 or earlier

## Tests
Covered by existing acceptance tests

Author: andrewnester

NEXT_CHANGELOG.md

@@ -3,12 +3,18 @@
 ## Release v0.266.0
 
 ### Notable Changes
+* Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or
+pipeline definition location instead of the configuration file location. Previously, the CLI would show a
+warning and fallback to resolving the path relative to the resource location. Users must update their bundle
+configurations to define all relative paths relative to the configuration file where the path is specified.
+See more details here: ([#3225](https://github.com/databricks/cli/pull/3225))
 * Add support volumes in Python support ([#3383])(https://github.com/databricks/cli/pull/3383))
 
 ### CLI
 
 ### Dependency updates
 
 ### Bundles
+* [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](https://github.com/databricks/cli/pull/3225))
 
 ### API Changes

acceptance/bundle/paths/fallback/output.txt

@@ -5,14 +5,36 @@ Warning: required field "entry_point" is not set
   in resources/my_job.yml:33:13
      override_job.yml:29:17
 
-Warn: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
-Warn: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
+Error: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:9:32
+     resources/my_job.yml:9:28
+
+Error: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:13:30
+     resources/my_job.yml:14:26
+
+Error: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:17:36
+     resources/my_job.yml:19:32
+
+Error: path ../src/sql.sql is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:24:25). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:24:25
+     resources/my_job.yml:27:21
+
+Error: path ../dist/wheel2.whl is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:33:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:33:24
+
+Error: path ../target/jar2.jar is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:41:24). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_job.yml:41:24
+
+Error: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:13:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_pipeline.yml:13:23
+
+Error: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).
+  in override_pipeline.yml:10:23
+
+
+Exit code: 1
 
 >>> [CLI] bundle validate -t error
 Warning: required field "entry_point" is not set

acceptance/bundle/paths/fallback_metric/databricks.yml

@@ -2,13 +2,15 @@ package mutator
 
 import (
 	"context"
+	"fmt"
 	"path/filepath"
 	"slices"
 
 	"github.com/databricks/cli/bundle/config/mutator/paths"
 
+	"github.com/databricks/cli/libs/diag"
 	"github.com/databricks/cli/libs/dyn"
-	"github.com/databricks/cli/libs/log"
+	"github.com/databricks/cli/libs/logdiag"
 )
 
 func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) {
@@ -45,8 +47,7 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value
 			return nv, nil
 		}
 
-		// If we failed to rewrite the path, try to rewrite it relative to the fallback directory.
-		// We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths].
+		// If we failed to rewrite the path, it uses an old path format which relied on fallback.
 		if fallback[key] != "" {
 			dir, nerr := locationDirectory(v.Location())
 			if nerr != nil {
@@ -66,8 +67,11 @@ func (t *translateContext) applyJobTranslations(ctx context.Context, v dyn.Value
 			originalValue := dyn.NewValue(originalPath, v.Locations())
 			nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts)
 			if nerr == nil {
-				t.b.Metrics.AddBoolValue("is_job_path_fallback", true)
-				log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location())
+				logdiag.LogDiag(ctx, diag.Diagnostic{
+					Severity:  diag.Error,
+					Summary:   fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()),
+					Locations: v.Locations(),
+				})
 				return nv, nil
 			}
 		}

acceptance/bundle/paths/fallback_metric/out.test.toml

@@ -2,12 +2,14 @@ package mutator
 
 import (
 	"context"
+	"fmt"
 	"path/filepath"
 
 	"github.com/databricks/cli/bundle/config/mutator/paths"
 
+	"github.com/databricks/cli/libs/diag"
 	"github.com/databricks/cli/libs/dyn"
-	"github.com/databricks/cli/libs/log"
+	"github.com/databricks/cli/libs/logdiag"
 )
 
 func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.Value) (dyn.Value, error) {
@@ -30,8 +32,7 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.
 			return nv, nil
 		}
 
-		// If we failed to rewrite the path, try to rewrite it relative to the fallback directory.
-		// We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths].
+		// If we failed to rewrite the path, it uses an old path format which relied on fallback.
 		if fallback[key] != "" {
 			dir, nerr := locationDirectory(v.Location())
 			if nerr != nil {
@@ -51,8 +52,11 @@ func (t *translateContext) applyPipelineTranslations(ctx context.Context, v dyn.
 			originalValue := dyn.NewValue(originalPath, v.Locations())
 			nv, nerr := t.rewriteValue(ctx, p, originalValue, fallback[key], opts)
 			if nerr == nil {
-				t.b.Metrics.AddBoolValue("is_pipeline_path_fallback", true)
-				log.Warnf(ctx, "path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.", originalPath, fallback[key], v.Location())
+				logdiag.LogDiag(ctx, diag.Diagnostic{
+					Severity:  diag.Error,
+					Summary:   fmt.Sprintf("path %s is defined relative to the %s directory (%s). Please update the path to be relative to the file where it is defined or use earlier version of CLI (0.261.0 or earlier).", originalPath, fallback[key], v.Location()),
+					Locations: v.Locations(),
+				})
 				return nv, nil
 			}
 		}