Skip to content

Commit 4d46dd1

Browse files
authored
DAB unable to validate container local file:///path-to-jar (#3892)
_##_ Changes In a runtime container, I need to be able to reference files. This PR allows the `file:///` path (which is valid in databricks templates) to do that. Currently the CLI attempts to resolve this path when it should now. Open to feedback here. ## Why When running a containerized job, I have pre-bundled assets that need to be referenced. If I set the path to `file:///` manually via the databricks UX, my job works. If I attempt to do it via the cli, I have this rendering issue. ``` databricks bundle deploy --target dev Error: file doesn't exist file:///opt/spark/jars/app.jar ``` With this change, the asset bundle is deployable. ## Tests Unit testing and manual deployment verification. <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. --> Signed-off-by: sshpuntoff <[email protected]>
1 parent 52a3fcf commit 4d46dd1

File tree

3 files changed

+18
-9
lines changed

3 files changed

+18
-9
lines changed

NEXT_CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,6 @@
99
### Dependency updates
1010

1111
### Bundles
12+
* Allow `file://` URIs in job libraries to reference runtime filesystem paths (e.g., JARs pre-installed on clusters via init scripts). These paths are no longer treated as local files to upload. ([#3884](https://github.com/databricks/cli/pull/3884))
1213

1314
### API Changes

bundle/libraries/local_path.go

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,17 +16,18 @@ import (
1616
// - myfile.txt
1717
// - ./myfile.txt
1818
// - ../myfile.txt
19-
// - file:///foo/bar/myfile.txt
2019
//
2120
// The following paths are considered remote:
2221
//
22+
// - file:///opt/spark/jars/myfile.jar
2323
// - dbfs:/mnt/myfile.txt
2424
// - s3:/mybucket/myfile.txt
2525
// - /Users/[email protected]/myfile.txt
2626
func IsLocalPath(p string) bool {
27-
// If the path has the explicit file scheme, it's a local path.
27+
// If the path has the file:// scheme, it's a runtime path (remote).
28+
// Users should use relative paths without scheme for local files to upload.
2829
if strings.HasPrefix(p, "file://") {
29-
return true
30+
return false
3031
}
3132

3233
// If the path has another scheme, it's a remote path.

bundle/libraries/local_path_test.go

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,30 @@ import (
88
)
99

1010
func TestIsLocalPath(t *testing.T) {
11-
// Relative paths, paths with the file scheme, and Windows paths.
11+
// Relative paths and Windows paths (local files to upload).
1212
assert.True(t, IsLocalPath("some/local/path"))
1313
assert.True(t, IsLocalPath("./some/local/path"))
14-
assert.True(t, IsLocalPath("file://path/to/package"))
1514
assert.True(t, IsLocalPath("C:\\path\\to\\package"))
1615
assert.True(t, IsLocalPath("myfile.txt"))
1716
assert.True(t, IsLocalPath("./myfile.txt"))
1817
assert.True(t, IsLocalPath("../myfile.txt"))
19-
assert.True(t, IsLocalPath("file:///foo/bar/myfile.txt"))
2018

21-
// Absolute paths.
19+
// Absolute paths without scheme (remote).
2220
assert.False(t, IsLocalPath("/some/full/path"))
2321
assert.False(t, IsLocalPath("/Workspace/path/to/package"))
2422
assert.False(t, IsLocalPath("/Users/path/to/package"))
2523

26-
// Paths with schemes.
24+
// file:// URIs are runtime paths (remote - not uploaded).
25+
assert.False(t, IsLocalPath("file:///foo/bar/myfile.txt"))
26+
assert.False(t, IsLocalPath("file:///opt/spark/jars/driver.jar"))
27+
assert.False(t, IsLocalPath("file:///"))
28+
assert.False(t, IsLocalPath("file:///absolute/path"))
29+
assert.False(t, IsLocalPath("file://path/to/package"))
30+
assert.False(t, IsLocalPath("file://foo/bar/myfile.txt"))
31+
assert.False(t, IsLocalPath("file://./relative.jar"))
32+
assert.False(t, IsLocalPath("file://../lib/package.whl"))
33+
34+
// Paths with other schemes (remote).
2735
assert.False(t, IsLocalPath("dbfs://path/to/package"))
2836
assert.False(t, IsLocalPath("dbfs:/path/to/package"))
2937
assert.False(t, IsLocalPath("s3://path/to/package"))
@@ -47,7 +55,6 @@ func TestIsLibraryLocal(t *testing.T) {
4755
{path: ".\\..\\local\\*.whl", expected: true},
4856
{path: "../../local/*.whl", expected: true},
4957
{path: "..\\..\\local\\*.whl", expected: true},
50-
{path: "file://path/to/package/whl.whl", expected: true},
5158
{path: "local/foo-bar.whl", expected: true},
5259

5360
{path: "", expected: false},

0 commit comments

Comments
 (0)