Skip to content

Commit f028ef0

Browse files
authored
acc: Add a test to deploy a single node cluster config in pipelines (#3951)
## Changes Add a test to deploy a single node cluster config in pipelines ## Why There's a difference in payload for this dabs configuration between direct and TF implementations
1 parent 25b3640 commit f028ef0

File tree

10 files changed

+115
-0
lines changed

10 files changed

+115
-0
lines changed
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
bundle:
2+
name: num-workers-zero
3+
4+
resources:
5+
pipelines:
6+
my:
7+
name: test-pipeline
8+
libraries:
9+
- file:
10+
path: "./foo.py"
11+
clusters:
12+
- node_type_id: i3.xlarge
13+
spark_conf:
14+
"spark.databricks.cluster.profile": "singleNode"
15+
"spark.master": "local[*]"
16+
custom_tags:
17+
"ResourceClass": "SingleNode"

acceptance/bundle/resources/pipelines/num-workers-zero/foo.py

Whitespace-only changes.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
2+
>>> errcode jq -s .[] | select(.method == "POST" and (.path | contains("/pipelines"))) out.requests.txt
3+
{
4+
"method": "POST",
5+
"path": "/api/2.0/pipelines",
6+
"body": {
7+
"channel": "CURRENT",
8+
"clusters": [
9+
{
10+
"custom_tags": {
11+
"ResourceClass": "SingleNode"
12+
},
13+
"label": "default",
14+
"node_type_id": "[NODE_TYPE_ID]",
15+
"spark_conf": {
16+
"spark.databricks.cluster.profile": "singleNode",
17+
"spark.master": "local[*]"
18+
}
19+
}
20+
],
21+
"deployment": {
22+
"kind": "BUNDLE",
23+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/state/metadata.json"
24+
},
25+
"edition": "ADVANCED",
26+
"libraries": [
27+
{
28+
"file": {
29+
"path": "/Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/files/foo.py"
30+
}
31+
}
32+
],
33+
"name": "test-pipeline"
34+
}
35+
}
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
2+
>>> errcode jq -s .[] | select(.method == "POST" and (.path | contains("/pipelines"))) out.requests.txt
3+
{
4+
"method": "POST",
5+
"path": "/api/2.0/pipelines",
6+
"body": {
7+
"channel": "CURRENT",
8+
"clusters": [
9+
{
10+
"custom_tags": {
11+
"ResourceClass": "SingleNode"
12+
},
13+
"label": "default",
14+
"node_type_id": "[NODE_TYPE_ID]",
15+
"num_workers": 0,
16+
"spark_conf": {
17+
"spark.databricks.cluster.profile": "singleNode",
18+
"spark.master": "local[*]"
19+
}
20+
}
21+
],
22+
"deployment": {
23+
"kind": "BUNDLE",
24+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/state/metadata.json"
25+
},
26+
"edition": "ADVANCED",
27+
"libraries": [
28+
{
29+
"file": {
30+
"path": "/Workspace/Users/[USERNAME]/.bundle/num-workers-zero/default/files/foo.py"
31+
}
32+
}
33+
],
34+
"name": "test-pipeline"
35+
}
36+
}

acceptance/bundle/resources/pipelines/num-workers-zero/out.test.toml

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

acceptance/bundle/resources/pipelines/num-workers-zero/output.txt

Whitespace-only changes.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
trace $CLI bundle deploy > out.deploy.$DATABRICKS_BUNDLE_ENGINE.txt 2>&1
2+
trace errcode jq -s '.[] | select(.method == "POST" and (.path | contains("/pipelines")))' out.requests.txt > out.requests.$DATABRICKS_BUNDLE_ENGINE.txt 2>&1
3+
rm out.requests.txt
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Badness = "num_workers value 0 is not sent in the direct deployment"
2+
Local = true
3+
Cloud = false
4+
RecordRequests = true
5+
6+
[EnvMatrix]
7+
DATABRICKS_BUNDLE_ENGINE = ["terraform", "direct"]

0 commit comments

Comments
 (0)